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Abstract 

With  advanced  autonomy,  Unmarmed  Aerial  Vehicle  (UAV)  operations  will 
likely  be  conducted  by  single  operators  controlling  multiple  UAVs.  As  operator 
attention  is  divided  across  multiple  supervisory  tasks,  there  is  a  need  to  support  the 
operator's  awareness  of  the  state  of  the  tasks  for  safe  and  effective  task  management. 
This  research  explores  enhancing  audio  cues  of  UAV  interfaces  for  this  futuristic  control 
of  multiple  UAVs  by  a  single  operator.  This  thesis  specifically  assesses  the  value  of 
continuous  and  discrete  audio  cues  as  indicators  of  course-deviations  or  late-arrivals  to 
targets  for  UAV  missions  with  single  and  multiple  UAVs.  In  particular,  this  thesis 
addresses  two  questions:  (1)  when  compared  with  discrete  audio,  does  continuous 
audio  better  aid  human  supervision  of  UAV  operations,  and  (2)  is  the  effectiveness  of 
the  discrete  or  continuous  audio  support  dependent  on  operator  workload? 

An  experiment  was  carried  out  on  the  Multiple  Autonomous  Unmarmed  Vehicle 
Experiment  (MAUVE)  test  bed  with  44  military  participants.  Specifically,  two 
continuous  audio  alerts  were  mapped  to  two  human  supervisory  tasks  within  MAUVE. 
These  continuous  alerts  were  tested  against  single  beep  discrete  alerts.  The  results 
show  that  the  use  of  the  continuous  audio  alerts  enhances  a  single  operator's 
performance  in  monitoring  single  and  multiple,  semi-autonomous  vehicles.  The  results 
also  emphasize  the  necessity  to  properly  integrate  the  continuous  audio  with  other 
auditory  alarms  and  visual  representations  in  a  display,  as  it  is  possible  for  discrete 
audio  alerts  to  be  masked  by  continuous  audio,  leaving  operators  reliant  on  the  visual 
aspects  of  the  display. 
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1.  Introduction 


The  use  of  Unmanned  Aerial  Vehicles  (UAVs)  is  growing  in  the  military,  federal, 
and  civil  domains.  Worldwide,  in  the  next  8  years  UAVs  will  be  a  15.7  billion  dollar 
industry,  where  in  the  United  States  alone  the  plan  is  to  have  9,420  mini  UAVs,  775 
tactical  UAVs,  570  minimum  altitude,  long  endurance  (MALE)  UAVs  and  60  Global 
Hawks  (Tsach,  Peled,  Penn,  Keshales,  &  Guedj,  2007).  In  fall  2006,  over  40  countries 
were  operating  UAVs  and  80  types  of  UAVs  were  in  existence,  with  the  United  States 
operating  3,000  UAVs  and  the  North  Atlantic  Treaty  Organization  (NATO)  operating 
3,600  UAVs  (Culbertson,  2006).  UAVs  are  not  just  in  demand  on  the  battlefield;  other 
government  departments  seeking  the  use  of  UAVs  include  the  Department  of 
Homeland  Security  and  the  U.S.  Coast  Guard  for  law  enforcement  and  border  patrol 
(DOD,  2007).  UAVs  are  also  sought  after  in  support  of  humanitarian  relief  efforts,  such 
as  with  flood  regions  in  the  United  States.  In  Missouri  in  May  2007,  Air  Force  Predator 
UAVs  were  on  standby  to  offer  assistance  in  the  flooding  recovery  efforts  (Arana- 
Barradas,  2007). 

In  his  spring  2008  speech  to  Air  Force  officers  at  Air  University,  Defense 
Secretary  Robert  M.  Gates  pointed  to  the  fact  that  within  the  Department  of  Defense 
(DOD)  there  has  been  a  25-fold  increase  in  UAV  operations  since  2001.  He  then  went 
on  to  say  that  this  increase  is  not  enough.  To  support  the  troops  in  Afghanistan  and 
Iraq,  work  must  be  done  to  further  integrate  UAVs  into  the  force  and  operations  (Gates, 
2008). 

One  of  the  efforts  to  further  integrate  UAVs  into  the  force  is  to  maximize  the 
human-to-UAV  ratio  as  a  "force  multiplier."  In  fall  2006,  General  William  T.  Hobbins, 
in  command  of  the  Allied  Air  Component  Command  and  European  Command  Air 
Component,  publicly  stated  that  part  of  the  solution  to  the  growing  UAV  demand  will 
be  to  have  a  single  operator  controlling  multiple  UAVs  (Culbertson,  2006),  because  a 
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single  operator  is  more  efficient  for  cost  and  operational  tempo  (Barbato,  Feitshans, 
Williams,  &  Fiughes,  2003;  Tsach  et  al.,  2007). 

To  achieve  force  multiplication,  humans  will  need  to  perform  a  supervisory  role 
as  vehicles  become  more  autonomous,  instead  of  attending  to  low-level  tasks  like 
manually  flying  the  aircraft.  One  way  to  help  humans  efficiently  and  safely  execute  this 
supervisory  task  is  to  maximize  the  use  of  each  sensory  channel  to  convey  information. 
In  aviation,  pilots  use  visual  and  auditory  senses  when  flying  an  aircraft.  A  unique 
aspect  of  UAV  supervision,  which  occurs  remotely,  is  that  if  the  operator  requires 
natural  signals  that  occur  in  maimed  aviation,  the  signals  must  be  synthetically  created. 
UAV  displays  are  predominantly  visual  displays  and  are  typically  mounted  in  mobile 
trailers,  truck  beds,  or  backpacks.  With  advancing  technology,  there  are  new, 
unexplored  ways  to  provide  display  information  across  multiple  sensory  charmels. 

When  there  is  an  overload  of  information,  utilizing  different  and  multiple 
sensory  charmels  can  effectively  increase  the  amount  of  information  processed 
(Wickens,  Lee,  Liu,  &  Becker,  2004).  When  using  multiple  sensory  charmels  (or  modes), 
certain  forms  of  information  are  better  conveyed  through  an  audio  presentation,  while 
others  are  better  conveyed  through  a  visual  presentation  (Deatherage,  1972;  Sorkin, 
1987).  Table  1-1  provides  a  list  of  known  benefits  for  the  visual  and  audio  channel 
presentations.  A  notable  benefit  for  auditory  presentation  is  its  effectiveness  at 
representing  simple  events  occurring  in  time  and  requiring  immediate  action,  as 
opposed  to  complex  events  occurring  at  a  location  in  space  and  not  requiring 
immediate  action.  Further,  because  of  its  omnipresence,  audio  is  usually  the  preferred 
sense  for  presenting  a  warning  (Simpson  &  Williams,  1980;  Sorkin,  1987).  Thus,  the 
audio  channel  is  effective  in  warning  of  potential  problems. 
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Table  1-1:  Auditory  Versus  Visual  Presentations  (Deatherage,  1972;  Sorkin,  1987). 


Use  auditory  presentation  if: 

1.  The  message  is  simple. 

2.  The  message  is  short. 

3.  The  message  will  not  be  referred  to  later. 

4.  The  message  deals  with  events  in  time. 

5.  The  message  calls  for  immediate  action. 

6.  The  visual  system  of  the  person  is 
overburdened. 

7.  The  receiving  location  is  too  bright  or  dark. 

8.  The  person's  job  requires  him  to  move  about 
continually. 


Use  visuai  presentation  if: 

1.  The  message  is  complex. 

2.  The  message  is  long. 

3.  The  message  will  be  referred  to  later. 

4.  The  message  deals  with  location  in  space. 

5.  The  message  does  not  call  for  immediate  action. 

6.  The  auditory  system  of  the  person  is 
overburdened. 

7.  The  receiving  location  is  too  noisy. 

8.  The  person's  job  allows  him  to  remain  in  one 
position. 


In  addition  to  information  presentation  modality,  another  concern  is  what 
specific  information  is  conveyed.  When  placed  in  a  supervisory  control  role,  a  human 
operator  may  have  to  respond  to  an  interruption  to  a  primary  task  of  monitoring.  To 
respond,  the  operator  needs  an  understanding  of  what  has  occurred  in  the  system, 
either  as  an  output  of  an  action  or  as  a  result  of  an  unseen  change  (Scott,  Mercier, 
Cummings,  &  Wang,  2006;  St.John,  Smallman,  &  Manes,  2005).  A  potential  way  to 
support  operator  understanding  of  a  monitored  task's  state  is  to  continuously  present 
information  so  that  the  operator  can  immediately  determine  a  task's  current  state,  as 
well  as  projected  future  states. 

The  objective  of  this  research  is  to  explore  ways  to  combine  audio  displays  with 
visual  displays  to  support  supervisory  tasks.  In  particular,  this  research  focuses  on 
comparing  continuous  to  discrete  audio  displays  to  understand  the  effects  of  a  constant 
versus  discrete  presentation  of  information.  Discrete  audio  displays  play  an  alert  once 
for  about  a  second  when  a  monitored  task  exceeds  limits.  In  contrast,  continuous  audio 
displays  are  audio  displays  that  always  indicate  some  system  state.  Chapter  2, 
Background,  will  further  frame  the  context  of  the  research  of  this  thesis. 
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1.1.  Problem  Statement 


The  primary  questions  addressed  through  this  research  are: 

1.  When  compared  with  discrete  audio,  does  continuous  audio  better  aid  human 
supervision  of  UAV  operations? 

2.  Is  the  effectiveness  of  the  discrete  or  continuous  audio  support  dependent  on 
operator  workload? 

Three  steps  are  followed  in  this  thesis  to  address  the  research  questions:  (1)  a 
multiple-UAV  simulator,  the  Multiple  Aerial  Unmanned  Vehicle  Experiment  (MAUVE) 
test  bed,  is  selected,  (2)  audio  displays  are  developed  and  integrated  into  this  multiple- 
UAV  simulator,  and  (3)  human  operator  performance  is  tested  in  the  MAUVE 
environment  to  compare  continuous  and  discrete  audio  displays. 

1.2.  Thesis  Organization 

This  thesis  is  organized  into  seven  chapters: 

•  Chapter  1,  Introduction,  provides  the  motivation  of  this  research,  research 
questions,  and  the  research  objectives  of  this  thesis. 

•  Chapter  2,  Background,  reviews  the  current  Unmanned  Aerial  System  (UAS) 
environment  and  the  needs  of  the  human  operator  supervising  UAS  operations. 
The  chapter  then  frames  the  research  of  this  thesis  in  terms  of  meeting  the  needs 
of  human  operators  supervising  UAS  operations. 

•  Chapter  3,  Simulator  and  Interface  Design,  presents  details  of  MAUVE  and  the 
associated  tasks  operators  supervise  on  MAUVE.  The  chapter  then  defines  the 
four  audio  alerts  tested  in  this  experiment  with  a  technical  description  of  the 
alerts  and  their  functions. 
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•  Chapter  4,  Method,  presents  the  research  questions  of  the  experiment,  the 
experimental  apparatus,  the  experimental  task,  the  participants,  the  independent 
and  dependent  variables,  and  the  data  collection. 

•  Chapter  5,  Results,  discusses  the  statistical  analysis  results.  The  chapter  also 
provides  a  description  of  the  participants'  post-test  subjective  feedback. 

•  Chapter  6,  Discussion,  synthesizes  the  applicable  lessons  that  can  be  extrapolated 
from  the  experimental  results. 

•  Chapter  7,  Conclusion,  reviews  the  answers  to  the  research  questions,  discusses 
continuous  audio  integration  in  the  real  world,  and  suggests  areas  for  future 
research. 
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2.  Background 


This  chapter  highlights  the  current  shift  toward  unmanned  systems  and  the 
needs  this  shift  generates  for  controlling  these  unmanned  systems.  It  presents  attention 
theories  linked  to  possible  solutions  for  aiding  human  operators  in  supervising 
unmanned  systems.  Further,  this  continuous  audio  research  is  framed  in  the  context  of 
previous  research  in  audio  display  support  of  supervisory  tasks.  Finally,  a  discussion 
of  the  research  hypotheses  and  the  groundings  of  these  hypotheses  closes  the  chapter. 

2.1.  U AS  Evolution 

2.1.1.  The  Current  Push 

Senior  military  officials  see  the  need  for  integrating  more  UAV  operations  in 
support  of  the  troops  on  the  ground  in  Iraq  and  Afghanistan  (USAF,  2007;  Whitney, 
2007).  Four  star  Combatant  Commanders  (COCOMs)  highly  desire  unmarmed  systems 
for  the  many  roles  these  systems  can  play  (Sullivan,  2008;  DOD,  2007).  Major 
advantages  of  unmarmed  systems  are  that  they  are  cheaper  and  have  more  endurance 
than  manned  aircraft  (Barbato  et  al.,  2003;  Gates,  2008).  Many  of  the  integral  roles  that 
COCOMs  may  envision  for  UAVs  are  encapsulated  in  recent  work  by  Nehme,  Crandall, 
and  Cummings  (2007),  which  includes  a  taxonomy  of  all  the  current  and  potential  roles 
that  UAVs  can  perform  with  today's  technology  (Figure  2-1). 

Unmarmed  vehicles  are  changing  how  warfighting  is  conducted,  and  they  are 
vital  to  providing  intelligence  and  reconnaissance  for  troops  on  the  ground.  As  of 
October  2006,  the  DOD  has  used  hand-launched  UAVs  to  fly  over  400,000  hours  of 
support  missions  for  Operation  Enduring  Freedom  and  Operation  Iraqi  Freedom  (DOD, 
2007).  The  Navy  used  UAVs  in  the  first  Gulf  War,  with  Pioneer  UAVs  performing 
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Figure  2-1:  Possible  UAV  Missions  (Nehme  et  al.,  2007). 
reconnaissance  missions  (Banks,  2000).  Currently,  the  Navy  is  developing  a  vertical- 
take-off-and-landing  tactical  UAV  (VTUAV),  designated  the  MQ-8B  Fire  Scout.  The 
Navy  plans  to  have  Fire  Scout  operational  and  throughout  the  fleet  in  fiscal  year  2009 
(Schroeder,  2008).  The  Army  has  400  UAVs  in  theater  for  the  Iraq  and  Afghanistan 
wars  (Sullivan,  2008).  Currently,  the  Army  has  785  UAVs,  comprised  of  Raven, 
Shadow,  Flunter,  and  Warrior  systems,  with  a  proposed  increase  to  4,755  UAVs  by  2023 
(Sullivan,  2008).  The  Air  Force,  too,  is  transitioning  toward  the  use  of  unmanned 
vehicle  technology.  Since  2001,  the  Air  Force  has  reduced  fighter  inventory  by  152 
aircraft,  while  simultaneously  increasing  UAS  platforms  by  113,  with  intentions  of 
continuing  this  over  the  next  several  years  (Randolph,  2007). 

There  is  a  need  for  automation  and  technology  to  better  support  human 
involvement  in  managing  these  unmarmed  systems.  From  January  to  October  2007,  the 
number  of  hours  for  mission  sorties  doubled  for  UASs  in  the  Air  Force,  creating  a 
manning  crisis.  Near  the  end  of  2007,  the  Air  Force  shifted  120  pilots  out  of  the  manned 
cockpit  to  the  ground  control  stations  for  UASs  (Staff,  2008).  A  technological  solution 
proposed  by  many  DOD  senior  leaders  and  researchers  to  alleviate  this  manning  crisis 
is  to  move  from  a  team  of  operators  controlling  a  UAV  to  having  a  single  operator 
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controlling  multiple  UAVs  (Barbato  et  al.,  2003;  Culbertson,  2006;  Tsach  et  al.,  2007). 

Figure  2-2  illustrates  this  transition. 

Team  of  people  flying  One  person  supervising 

one  UAV  mission  of  multiple  UAVS 

IHM  —  t 


Figure  2-2:  Transition  to  Multiple-UAV  Supervision. 


2.1.2.  The  Human  in  the  System 

The  two  terms  UAV  and  UAS  may  appear  interchangeable.  However,  there  is  a 
key  distinction  between  them.  The  Office  of  the  Secretary  of  Defense  has  issued  the 
following  definition: 

Unmanned  Vehicle.  A  powered  vehicle  that  does  not  carry  a  human  operator, 
can  be  operated  autonomously  or  remotely,  can  be  expendable  or  recoverable, 
and  can  carry  a  lethal  or  nonlethal  payload. .  .Unmanned  vehicles  are  the  primary 
component  of  unmanned  systems  (DOD,  2007). 

The  key  point  is  that  the  UAV  is  a  sub-component  that  helps  comprise  the  overall  UAS. 
Another  key  sub-component  to  the  UAS  is  the  human  operator  controlling  UAV 
operations  through  a  ground  control  station. 

The  term  unmanned  does  not  imply  removing  the  human  completely  because  a 
human  is  still  involved  in  unmanned  operations,  even  in  the  most  autonomous 
situation.  The  human's  role  is  not  eliminated  just  because  the  human  is  no  longer  co¬ 
located  with  the  mission  (DOD,  2007).  Even  as  autonomy  progresses,  the  need  for 


19 


human  reasoning  and  judgment  will  never  be  replaced  within  the  UAS  (Economist, 
2007).  Furthermore,  the  DOD  has  stated  that  humans  are  still  necessary  "to  interpret 
sensor  information,  monitor  systems,  diagnose  problems,  coordinate  mission  time  lines, 
manage  consumables  and  other  resources,  authorize  the  use  of  weapons  or  other 
mission  activities,  and  maintain  system  components"  (DOD,  2007).  Thus  any  UAS  must 
be  designed  to  support  the  role  of  the  human  operator  in  establishing  the  system's  goal, 
supervising  the  system  to  guide  it  toward  the  goal,  and  ensuring  mission  success 
(Barbato  et  al.,  2003;  DOD,  2007).  The  human  operator's  supervisory  role  is  essential  to 
UAS  operations.  This  thesis  explores  areas  in  which  new  technology  can  support  the 
single,  human  operator  in  supervising  the  operations  of  multiple  UAVs. 

2.1.3.  The  UAV  Ground  Control  Station 

To  support  the  supervision  of  UAS  operations,  an  understanding  of  the  current 
UAV  ground  control  stations  is  needed.  Today,  the  Air  Force  and  Army  operate  most 
UAV  missions  from  trailers  or  the  backs  of  trucks.  Most  operations  rely  primarily  on 
visual  displays.  In  the  Air  Force,  the  solution  for  providing  additional  missing 
information  to  UAS  operators  has  been  to  add  more  visual  displays.  However,  humans 
have  a  limited  capacity  of  resources  to  process  information,  so  adding  more  visual 
displays  will  only  further  overload  operators  (Wickens  et  al.,  2004).  One  solution 
explored  by  this  research  to  help  humans  process  more  information  is  adding  audio 
displays  that  can  draw  upon  more  attentional  resources  without  adding  significantly  to 
the  workload. 

Operating  in  field  conditions,  UAS  ground  control  environments  can  be  very 
noisy  and  aurally  cumbersome,  with  generators  rurming  in  the  background  and 
required  communication  between  inter  and  intra  team  members.  This  is  significant  to 
note  when  considering  auditory  displays.  Any  implementation  of  an  audio  alert  in 
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these  types  of  field  conditions  will  have  other  audio  alerts  competing  for  attention  and 
only  certain  available  frequencies  over  which  alerts  can  be  generated  and  heard. 

2.2.  Attention  Theories 

The  processes  by  which  humans  acquire  and  aggregate  information  for  proper 
decision  making  and  taking  action  rely  upon  attention  resources  (Parasuraman, 
Sheridan,  &  Wickens,  2000).  Understanding  these  processes  is  important  in  the  design 
of  UAV  supervisory  control  because  various  types  and  amounts  of  attention  are  needed 
for  the  multiple  tasks  performed  by  the  UAVs.  This  section  discusses  different  attention 
types  and  the  allocation  of  attention  resources  that  are  used  in  human  cognitive 
processing. 

2.2.1.  Selective,  Divided,  and  Sustained  Attention 

The  primary  task  for  an  operator  overseeing  multiple-UAV  operations  will  be  to 
simultaneously  monitor  particular  tasks  across  multiple  UAVs.  This  will  require  the 
UAV  operator  to  time-share  between  the  tasks  to  maintain  continual  awareness  and 
respond  at  the  appropriate  time  to  each  task.  Further,  the  missions  will  occur  over 
prolonged  periods  of  time,  during  which  the  human  operator  will  monitor  for  abnormal 
conditions  and  also  provide  high-level  directives  to  the  semi-autonomous  UAV 
operations.  Therefore,  it  is  assumed  that  in  the  supervisory  control  role,  UAV  operators 
will  primarily  perform  their  role  with  selective,  divided,  and  sustained  attention. 

Selective  attention  occurs  when  an  operator  must  "monitor  several  sources  of 
information  to  determine  whether  a  particular  event  has  occurred"  (Sanders  & 
McCormick,  1993).  An  example  of  selective  attention  in  multiple-UAV  operations  is 
when  a  sensor  operator  is  watching  multiple  UAV  video  feeds  to  locate  a  particular 
enemy  vehicle. 
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Divided  attention  occurs  when  "two  or  more  separate  tasks  must  be  performed 
simultaneously,  and  attention  must  be  paid  to  both"  (Sanders  &  McCormick,  1993). 
Continuing  with  the  same  example,  the  sensor  operator  may  be  forced  to  rely  on 
divided  attention  if  he  is  required  to  keep  track  of  a  vehicle  he  has  located,  as  well  as 
continuing  to  monitor  the  other  video  inputs  for  potential  enemy  vehicles  that  require 
some  action.  This  type  of  attention  requires  a  form  of  time-sharing,  where  the  operator 
splits  his  time  between  tasks. 

Sustained  attention  occurs  when  an  operator  "sustains  attention  over  prolonged 
periods  of  time,  without  rest,  in  order  to  detect  infrequently  occurring  signals" 
(Parasuraman,  Warm,  &  Dember,  1987;  Sanders  &  McCormick,  1993).  This  is  generally 
the  bulk  of  a  UAV  surveillance  mission.  The  sensor  operator  in  the  previous  example 
will  spend  a  majority  of  his  time  simply  watching  for  a  change  in  the  videos  that  would 
indicate  an  enemy  vehicle. 

2.2.2.  Attention  Resource  Theories 

To  maintain  a  high  level  of  performance,  an  operator  must  manage  his 
attentional  resources.  A  scarcity  of  resources  exists,  and  this  limited  pool  of  resources  is 
what  is  drawn  upon  by  the  various  attention  types  (Wickens  et  al.,  2004).  Three 
hypotheses  proposed  in  the  literature  (Hirst,  1986)  to  explain  attention  resources  are  as 
follows: 


1.  One  central  resource  -  a  single  resource  from  which  all  tasks  draw. 

2.  Multiple  resources  -  with  some  tasks  drawing  from  some  resource 
pools  and  not  others.  For  example,  verbal  tasks  draw  from  a  verbal 
resource  pool  and  not  a  visual  resource  pool. 

3.  Both  a  multiple  resources  pool  as  well  as  a  central  resource  pool. 
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In  general,  these  theories  assume  a  scarcity  of  resources.  The  single  resource 
theory  holds  that  there  is  a  single  pool  for  attention  resources  that  is  drawn  upon  for  all 
the  mental  processing  (Kahneman,  1973;  Moray,  1967).  It  proposes  that  humans  have 
limited  cognitive  resources  in  this  single  pool  and  that  when  some  of  these  resources 
are  allocated,  alternative  tasks  can  only  use  what  remains  unallocated.  There  are  some 
problems  with  the  single  resource  theory.  First,  when  a  task  is  actually  performed  on  a 
single  processing  code  (e.g.,  verbal,  spatial)  or  modality  (e.g.,  visual,  auditory),  more 
attention  resources  appear  to  be  required  than  when  the  task  is  performed  using 
multiple  processing  codes  or  modalities.  Second,  within  some  groupings  of  tasks,  if  the 
difficulty  of  one  task  increases,  there  appears  to  be  no  effect  on  the  other  tasks. 
However,  in  groupings  with  similar  tasks,  when  the  difficulty  of  one  task  increases,  it 
seems  to  affect  the  performance  on  the  other  tasks  (Sanders  &  McCormick,  1993; 
Wickens  &  Hollands,  2000).  There  appears  to  be  interference  and  failed  time-sharing 
within  certain  task  groupings,  but  not  others.  These  problems  indicate  that  perhaps 
there  are  separate  attentional  resource  pools  for  the  different  processing  codes  or 
modalities. 

2.2.3.  Multiple  Resource  Theory 

As  opposed  to  the  single  resource  theory,  the  multiple  resource  theory  proposes 
that  there  are  multiple  independent  pools  from  which  attentional  resources  are  drawn. 
As  a  result,  there  can  be  increased  efficiency  when  time-sharing  between  multiple  tasks 
(Sanders  &  McCormick,  1993).  Interference,  discussed  in  the  single  resource  theory 
above,  is  a  result  of  multiple  tasks  requiring  resources  from  the  same  pool.  Multiple 
resource  theory  propounds  that  information  can  be  processed  concurrently  by  dividing 
presentation  over  various  channels  so  that  the  information  is  processed  simultaneously 
by  different  pools  of  attentional  resources  (Wickens  et  al.,  2004). 
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Multiple  resource  theory  holds  that  the  effect  one  task  has  on  another  depends 
on  the  type  of  tasks.  An  example  is  when  someone  tries  rubbing  his  stomach  while 
patting  his  head.  This  is  difficult  because  he  has  run  out  of  attentional  resources,  while 
speaking  and  rubbing  one's  stomach  is  completely  possible  because  there  are  ample 
attentional  resources.  Wickens  and  Hollands  (2000),  in  their  model  in  Figure  2-3, 
explains  why.  Their  model  provides  four  dimensions  to  classify  which  resource  pools 
attentional  resources  are  coming  from:  stages,  input  modalities,  processing  codes,  and 
responses.  The  example  of  patting  one's  head  and  rubbing  one's  stomach  is  an 
illustration  of  two  manual  responses,  which  require  resources  from  the  same  pool. 
While  rubbing  one's  stomach  and  speaking,  respectively,  are  illustrations  of  manual 
and  vocal  responses.  Because  they  are  different  response  types,  they  have  different 
attentional  pools  of  resources  to  draw  upon. 

M -  Stages - ► 

Perception  Working  memory  Responding 


Figure  2-3:  Attentional  Resource  Pools  (Wickens  &  Hollands,  2000). 

The  focus  of  this  thesis  is  on  using  aural  and  visual  input  modalities  to  aid  in  the 
multiple-UAV  supervision  task.  Input  modalities  are  the  charmels  over  which 
information  may  be  presented  to  an  individual.  Research  has  shown  that  if  a  task's 
information  is  presented  in  dual  coding,  on  differing  modalities,  such  as  the  auditory 
and  visual  sensory  charmels,  time-sharing  is  more  efficient  (Miller,  1991).  In  contrast,  if 
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tasks  are  redundantly  coded,  on  the  same  modality,  such  as  the  auditory  and  auditory 
or  visual  and  visual  sensory  charmels,  interference  occurs  (Sanders  &  McCormick, 
1993).  Research  by  Streeter,  Vitello,  and  Wonsiewicz  (1985)  showed  human  operators 
can  better  process  aural  commands  for  driving,  while  the  drivers  are  focusing  their 
visual  attention  on  the  road. 

Previous  research  has  shown  that  dual  coding,  i.e.,  displaying  information  on 
two  separate  modalities  is  beneficial  (Miller,  1991;  Streeter  et  al.,  1985).  For  example,  it 
has  been  found  that  using  both  visual  and  audio  multimedia  in  unison  is  a  useful  way 
to  present  information  (Mayer,  1999).  Similarly,  it  should  be  beneficial  to  dual  code 
information  in  both  the  visual  and  audio  modalities  to  aid  operators  in  monitoring  UAV 
operations.  The  premise  of  this  research  is  that  by  dividing  the  monitoring  of  tasks  over 
multiple  modalities,  the  operator  will  better  be  able  to  integrate  all  attentional  resource 
pools  toward  efficiently  controlling  multiple  tasks.  In  fact,  this  is  not  a  new  idea.  In  a 
meta-analysis  of  43  previous  studies  (Burke  et  al.,  2006),  researchers  concluded  that  in 
UAV  operations  dual  coding  information  on  the  visual  and  auditory  or  visual  and 
tactile  modalities  increases  performance,  while  potentially  decreasing  workload.  One 
study  in  the  UAV  domain  compared  a  visual  display  with  a  visual  display 
supplemented  with  an  audio  alert  and  found  enhanced  performance  with  the  latter. 
This  same  study  also  showed  that  when  going  from  supervising  one  UAV  to 
supervising  two  UAVs,  operator  performance  decreased,  regardless  of  the  display 
(Dixon,  Wickens,  &  Chang,  2005).  This  thesis  explores  continuous  audio  alerts  which 
have  not  yet  been  combined  with  visual  displays  for  supporting  UAV  operators. 

2.3.  Audio  Research 

This  thesis  focuses  on  the  use  of  continuous  audio  alerts,  in  particular, 
sonifications,  in  a  multi-modal  display  to  help  an  operator  supervise  multiple  tasks 
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across  multiple  unmanned  aerial  vehicles.  The  existing  audio  literature  is  approached 
from  three  perspectives.  First,  audio  research  for  supervisory  control  with  multi-modal 
displays  is  reviewed  to  help  frame  what  types  of  supplemental  audio  displays  have  not 
been  researched  for  application  with  UAV  supervisory  tasks.  Second,  a  form  of  audio 
that  has  not  been  extensively  researched  in  its  ability  to  aid  supervisory  control  is 
continuous  audio.  For  this  reason,  a  review  of  applicable  continuous  audio  research  is 
completed.  Third,  a  form  of  continuous  audio  that  portrays  relations  in  information 
and  information  changes  is  sonifications.  In  closing  this  section,  a  review  of  key 
supervisory  control  sonifications  research  is  accomplished. 

2.3.1.  Supervisory  Control  with  Multi-Modal  Displays 

A  significant  body  of  audio  research  applicable  for  supervisory  control  of 
multiple  unmanned  vehicles  focuses  on  spatial  audio  (Nehme  &  Cummings,  2006). 
Spatial  audio  is  3-dimensional  (3D)  audio  in  which  audio  is  presented  so  that  specific 
signals  come  from  specific  locations  in  the  360-degree  range  around  someone's  head. 
An  example  is  an  audio  alert  for  one  UAV  presented  over  a  headset  directly  in  front  of 
the  operator,  while  alerts  for  another  UAV  are  cormected  to  signals  directly  behind  the 
operator. 

The  spatial  audio  research  applicable  to  supporting  operators  performing 
supervisory  tasks  has  primarily  focused  on  sound  localization  (Martin,  McAnally,  & 
Senova,  2001;  Wightman  &  Kistler,  1989),  detection  and  localization  of  visual  targets 
(Bolia,  D'Angelo,  &  McKinley,  1999;  Bronkhorst,  Veltman,  &  vanBreda,  1996;  Flanagan, 
McAnally,  Martin,  Meehan,  &  Oldfield,  1998;  Nelson  et  al.,  1998),  collision  avoidance 
(Begault  &  Pittman,  1996),  and  navigation  (Moroney  et  al.,  1999).  In  the  aviation 
domain,  this  research  has  primarily  been  applied  in  the  maimed  cockpit;  because  of  the 
supervisory  nature  of  these  tasks,  this  research  is  applicable  in  the  unmanned  cockpit  as 
well.  In  particular,  spatial  audio  has  been  shown  to  help  reduce  search  times  and 
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produce  faster  response  times  for  target  acquisition  (Parker,  Smith,  Stephan,  Martin,  & 
McAnally,  2004)  and  alert  and  warning  detection  (Begault  &  Pittman,  1996).  The 
literature  suggests  that  spatial  audio  helps  mitigate  visual  change  blindness  effects, 
which  occur  when  an  operator  is  so  focused  visually  on  one  task  that  the  operator's 
focused  attention  causes  another  visual  warning  to  be  missed  (Nehme  &  Cummings, 
2006).  Outside  of  spatial  audio,  little  other  audio  research  has  been  done  in  the 
supervisory  domain  of  multiple  UAVs.  In  combing  the  audio  literature,  little  applicable 
research  is  found  where  ambient  noise  headphones  and  associated  filtering  techniques 
are  used  to  support  a  supervisory  task  in  a  multi-modal  display  (Nehme  &  Cummings, 
2006).  A  form  of  this  audio  is  continuous  audio,  which  is  explored  next. 

2.3.2.  Continuous  Audio 

Continuous  audio  alerts  play  continually  to  map  information  on  the  state  of  a 
task.  The  contrast  to  continuous  audio  is  discrete  audio,  which  only  plays  at  a  discrete 
point  in  time  when  a  task's  state  changes.  Discrete  audio  can  be  used  to  aurally 
interrupt  an  operator  to  notify  him  of  a  problem.  Continuous  audio  is  essentially  a 
peripheral  monitoring  system,  which  allows  the  operator  to  constantly  monitor  for 
abnormal  situations;  in  this  case,  an  operator  is  using  divided  attention  to  time-share 
their  focus  between  performing  or  monitoring  multiple  tasks,  with  the  audio  as  a  cue 
guiding  when  to  shift  focus. 

Most  audio  alert  research  has  focused  on  discrete  alerts  in  the  forms  of  a  "beep" 
and  "ding"  (Brewster,  Wright,  &  Edwards,  1994).  However,  a  limited  number  of  formal 
human  studies  have  investigated  continuous  audio  alerts  (Pacey  &  MacGregor,  2001). 
The  continuous  audio  research  that  has  been  completed  has  focused  on  using 
continuous  audio  and  discrete  audio  alerts  to  enhance  a  visual  display  for  a  monitoring 
task.  Research  has  shown  the  addition  of  continuous  audio  with  a  visual  representation 
enhances  operator  performance  (Crease  &  Brewster,  1998;  Pacey  &  MacGregor,  2001). 
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The  following  paragraphs  will  discuss  the  details  of  these  experiments  that  have  been 
run  with  continuous  audio. 

One  of  the  first  experiments  run  with  human  participants  was  by  Crease  and 
Brewster  (1998).  Participants  completed  a  primary  task  of  typing  poems,  while 
monitoring  the  downloading  of  files  as  a  secondary  task.  Participants  had  either  a 
visual  progression  display  of  the  downloading  or  a  visual  and  auditory  progression 
display.  The  results  of  the  experiment  showed  that  participants  with  the  continuous 
audio  display  were  significantly  quicker  at  responding  than  the  participants  with  only 
the  visual  display.  This  is  an  example  of  how  displaying  the  information  on  multiple 
modalities  enhanced  operator  performance  for  a  monitoring  task. 

In  a  similar  experiment,  Pacey  and  MacGregor  (2001)  compared  a  baseline  visual 
display  with  the  visual  display  plus  one  of  three  different  audio  displays:  discrete 
chimes,  continuous  wind,  and  a  continuous  audio  progress  bar.  The  primary  difference 
between  the  audio  displays  was  the  amount  of  information  presented  by  the  display. 
The  discrete  chimes  presented  two  of  the  task's  properties,  i.e.,  when  a  task  began  and 
when  the  task  ended,  with  separate  discrete  chimes  that  played  when  an  event  began 
and  when  an  event  ended.  The  continuous  wind  was  a  low  wind  noise  that  played 
continually  from  when  a  task  began  until  it  ended.  It  added  a  third  piece  of 
information;  as  long  as  the  wind  was  playing,  the  operator  knew  that  the  task  was  still 
occurring.  The  continuous  audio  progress  bar  presented  six  of  the  task's  properties, 
such  as  scope,  initiation,  progress,  heartbeat,  remainder,  and  completion,  through 
continuous  audio  created  by  different  arrangements  of  12  different  notes;  this  audio 
played  continually  throughout  an  event.  While  the  results  were  mixed  dependent  upon 
what  performance  metrics  were  considered,  it  is  clear  that  adding  the  audio  alert  to  the 
visual  display  significantly  enhanced  the  participants'  ability  to  complete  the  primary 
and  secondary  tasks.  Pacey  and  MacGregor's  findings  showed  that  when  simply 
considering  the  reaction  times  for  catching  the  downloads  ending,  the  audio  progress 
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bar  was  significantly  better  than  the  chimes,  but  that  the  chimes  were  significantly 
better  than  the  wind  or  visual-only  display.  However,  as  indicated  by  the  primary  task 
metrics,  typing  speed  and  accuracy,  participants  using  the  chimes  performed 
significantly  better  than  the  auditory  progress  bar  or  visual-only  display.  The 
conclusion,  therefore,  is  that  the  chimes  provided  the  best  results  of  enhancing  the 
ability  to  perform  the  secondary  task  of  catching  download  endings,  without 
hampering  the  primary  task  of  typing  poetry. 

With  Pacey  and  MacGregor's  mixed  results  on  the  effects  of  discrete  versus 
continuous  audio  support  on  both  a  secondary  and  primary  task,  this  thesis  will  further 
explore  whether  monitoring  tasks  in  supervisory  control  is  better  supported  by  a  typical 
discrete  audio  alert  or  a  by  a  continuous  audio  alert.  Further,  this  thesis  will  explore 
using  both  discrete  audio  and  continuous  audio  in  a  single  audio  condition  to  support 
multiple  supervisory  tasks.  Because  of  the  limited  research  available  on  continuous 
versus  discrete  audio,  the  results  of  this  thesis  will  be  an  important  contribution  to  the 
audio  alert  research. 

2.3.3.  Supervisory  Control  Sonifications 

Continuous  audio  is  defined  above  as  audio  alerts  that  play  continually  to 
provide  continual  information  on  a  task's  state.  Sonifications  are  a  form  of  continuous 
audio,  in  which  data  and  relations  in  data  are  mapped  to  relations  in  the  audio  display 
(Kramer,  1994).  An  example  of  a  sonification  that  Tarmen  (1998)  gives  is  sonar  on  a 
submarine  that  indicates  the  distance  of  a  torpedo.  As  the  torpedo  gets  closer,  the  sonar 
beeps  become  more  intense  on  a  continual  scale.  The  intensity  of  the  beeps  provides 
relational  data  of  how  far  the  torpedo  is  from  the  submarine.  Sonifications  are  a  subset 
of  continuous  audio.  The  torpedo  example  is  sonification  in  particular,  because  the 
intensity  of  beeping  noise  provides  the  relational  data  of  how  far  the  torpedo  is  from 
the  submarine. 


29 


Research  has  shown  sonifications  to  be  beneficial  in  medical  and  assistive 


technologies,  scientific  visualizations,  engineering  analysis,  emergency  alerts,  and 
aircraft  cockpits  (Barrass  &  Kramer,  1999).  One  example  in  the  medical  domain  is  a 
study  by  Loeb  and  Fitch  (2002)  that  compared  three  types  of  anesthesiology  displays: 
visual  only,  auditory  only,  and  a  combined  visual  and  auditory  display.  The  task  for 
participants  was  to  monitor  if  any  of  six  vital  signs  exceeded  normal  limits.  For  the 
auditory  displays,  participants  received  independent  streams  of  audio  for  the  vital 
signs.  For  the  heartbeat,  the  audio  was  a  low-pitched,  repetitive,  thudding  to  resemble 
what  one  may  envision  the  heart  sounding  like.  For  the  respiratory  system,  a  higher- 
pitched  amplitude-modulated  signal  was  used.  For  the  remaining  four  vitals, 
variations  of  modulation  and  filtering  were  used  to  generate  the  audio.  The 
participants  with  the  combined  visual  and  audio  display  had  the  quickest  reaction 
times,  but  the  visual-only  display  produced  the  most  accurate  reactions. 

In  a  similar  anesthesiology  study,  Watson  and  Sanderson  (2004)  asked 
participants  to  divide  their  attention  between  monitoring  for  an  abnormal  event  and 
performing  some  other  primary  task.  Thus,  monitoring  was  a  secondary  task.  In 
particular,  they  tested  various  modalities  in  supporting  anesthesiologists  and  non¬ 
anesthesiologists  in  monitoring  a  respiratory  rate  as  the  secondary  task.  There  were 
three  displays  tested:  sonifications  only,  visual  only,  and  combined  sonifications  and 
visual.  The  sonifications  were  respiratory  sonifications  in  which  the  breathing  rate  was 
mapped  to  a  pure  tone  with  musical  notes  on  the  third  interval,  for  inhalation  and 
exhalation.  Watson  and  Sanderson  (2004)  results  illustrated  that  regardless  of  the 
display  modality,  the  performance  was  the  same  for  the  secondary  task  of  monitoring 
the  breathing,  but  that  for  the  time-shared  primary  task,  participants  performed  best 
with  the  combined  display.  These  results  show  the  benefit  of  the  combined  display  in 
helping  participants  time-share.  Flowever,  because  only  sonifications  were  used  as  the 
audio  support  in  these  experiments,  there  is  no  comparison  between  sonifications  and 
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discrete  audio.  This  comparison  is  what  this  thesis  will  explore.  Further,  these  results 
are  limited  in  that  they  only  tested  the  effect  of  the  displays  on  an  anesthesiologist 
monitoring  a  single  patient.  This  thesis  will  analyze  the  benefits  of  sonifications  while 
monitoring  multiple  tasks  (UAVs)  as  opposed  to  a  single  task. 

2.4.  Research  Hypotheses 

Having  framed  the  research  of  this  thesis,  two  hypotheses  are  proposed.  The 
first  hypothesis  is  that  in  comparison  with  a  discrete  audio  scheme,  continuous  audio, 
in  the  form  of  a  sonification,  improves  operator  performance  for  monitoring  sustained 
and  divided  attention  tasks  in  a  time-intensive  dynamic  environment.  Continuous 
audio  reduces  the  uncertainty  of  a  future  state  by  allowing  operators  to  monitor  trends 
in  a  changing  state  over  a  secondary  modality.  Further,  by  placing  this  information 
over  the  audio  charmel,  there  is  immediate  access  to  the  information.  Immediate 
information  access  is  important  to  performance  because  humans  tend  to  re-check 
signals  repeatedly  because  of  limited  human  memory  (Moray,  1981). 

The  second  hypothesis  is  that  the  benefits  of  continuous  audio  will  be  more 
beneficial  for  operators  controlling  multiple  vehicles  rather  than  a  single  vehicle,  since 
the  workload  under  the  multiple  case  is  significantly  higher.  Because  the  monitoring  of 
multiple  vehicles  increases  workload,  more  attentional  resources  are  required  from 
supervisors.  The  addition  of  the  audio  displays  to  the  visual  display  provides  an 
additional  pool  from  which  to  draw  attentional  resources  for  processing  some  of  the 
information  for  managing  the  UAVs.  Furthermore,  the  continuous  audio  allows  for 
more  information  to  be  presented  over  the  audio  sensory  channel  than  the  discrete 
audio  because  it  provides  a  continual  status.  Because  it  provides  more  information  than 
the  discrete  audio,  the  expectation  is  that  in  a  high  workload  situation,  where  the 
attentional  resources  are  fully  utilized,  the  additional  continuous  audio  will  allow 
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operators  to  draw  more  attentional  resources  from  the  audio  resource  pool,  which  will 
better  support  their  performance. 

To  test  these  hypotheses,  a  multiple-UAV  simulator  experiment  was  conducted, 
which  will  be  discussed  in  more  detail  in  Chapters  3  and  4. 
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3.  Simulator  and  Interface  Design 


The  Multiple  Aerial  Unmanned  Vehicle  Experiment  (MAUVE)  test  bed  was 
developed  in  previous  research  to  allow  human  operators  to  perform  supervisory 
control  tasks  as  they  monitor  the  progression  of  multiple  UAVs  arming  and  firing  on 
targets,  performing  battle  damage  assessment,  and  avoiding  threat  areas  (Mitchell, 
2005).  The  following  sections  will  present  how  MAUVE  was  customized  for  this 
experiment  and  the  form  of  the  audio  displays  that  were  created  and  integrated  into  the 
MAUVE  interface  for  this  experiment. 

3.1.  Multiple  Aerial  Unmanned  Vehicle  Experiment  (MAUVE)  Test  Bed 

MAUVE  is  a  two  screen  interface  that  allows  the  simulation  of  any  number  of 
UAVs  conducting  strike  operations  in  hostile  environments.  MAUVE,  for  this 
experiment,  allowed  the  operator  to  arm  and  fire  on  targets  with  various  priorities, 
while  monitoring  each  UAV's  flight  of  path,  event  timeline,  and  radio  traffic. 

The  MAUVE  test  bed  provided  a  map  for  geo-spatial  tracking  of  UAVs  on  their 
preset  flight  paths,  along  with  an  interaction  control  panel  for  making  UAV  control 
inputs  (Left  Display,  Figure  3-1).  Control  inputs  included  arming  and  firing  a  UAV  to 
destroy  targets,  directing  specific  path  changes,  and  acknowledging  radio  calls.  For 
each  control  input,  operators  simply  clicked  one  or  two  buttons  on  the  control  panel 
(Figure  3-2). 

A  timeline  for  each  UAV,  a  scheduling  decision  support  visualization  (DSV)  for 
each  UAV  (which  will  be  discussed  in  detail  in  a  subsequent  section),  and  chat 
interfaces  were  also  provided  (Right  Display,  Figure  3-1).  This  timeline  display  helped 
operators  ensure  UAVs  would  be  on  time  to  targets  by  monitoring  when  UAVs  would 
arrive  at  targets  and  in  what  phase  of  the  mission  they  were  currently  located.  In 
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Figure  3-1:  Multiple  Aerial  Unmanned  Vehicle  Experiment  (MAUVE)  Test  Bed. 
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Eigure  3-2:  UAV  Interaction  Control  Panel 
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Table  3-1:  UAV  Color-Coded  Flight  Phases 


UAV  Action 

Color 

Enroute 

Gray 

Loitering 

Blue 

Arming  Payload 

Yellow 

Firing  Payload 

Orange 

Return  to  Base 

Green 

particular,  the  timeline  used  the  color-coding  provided  in  Table  3-1  to  represent  the 
phase  of  flight. 

The  general  task  for  MAUVE  operators  for  this  experiment  was  to  monitor  one 
or  more  UAVs  on  a  preplarmed  mission  involving  several  waypoints  and  targets.  The 
primary  task  of  the  operator  was  to  ensure  weapons  were  dropped  on  the  correct 
targets  during  the  specified  windows  of  opportunity.  However,  unexpected  head  winds 
or  crosswinds  could  cause  UAVs  to  slow  their  speed  or  drift  off  course,  requiring  the 
operator  to  take  corrective  action  for  projected  target  late-arrivals  or  course-deviations. 
Late-arrivals  were  a  higher  priority  than  course-deviations.  In  addressing  late-arrivals, 
the  priority  of  the  target  (high,  medium,  or  low)  also  had  to  be  addressed,  because  a 
high  priority  target  was  more  important  than  a  medium  priority  target,  and  so  on.  In 
addition,  operators  also  had  to  ensure  the  UAVs  would  return  to  base  by  a  specified 
time. 

Figure  3-3  provides  the  visual  representation  for  a  late-arrival.  A  black  rectangle 
on  the  timeline  showed  when  the  UAV  was  going  to  arrive  at  the  labeled  target,  and  it 
changed  from  black  to  red  when  the  UAV  was  going  to  be  late.  This  could  also  be  seen 
when  the  black  rectangle  moved  past  the  orange  window  of  opportunity.  On  the 
timeline  below,  the  red  rectangle  labeled  "T-IL"  indicates  that  the  UAV  will  be  late  to 
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target  T-IL.  In  contrast,  the  black  rectangle  labeled  "T-2M"  indicates  that  the  UAV  will 
be  on  time  to  target  T-2M. 


L  WP-1B 

M 

r 

_ 

r - 

2 

Cl 

1-^ 

T-1L  T-2M 

Figure  3-3:  Late-arrival  Illustration. 

The  decision  support  visualization  (DSV)  was  an  additional  visual  tool  used 
when  a  UAV  was  late  to  a  target.  The  DSV  was  designed  to  illustrate  the  current  state 
of  a  UAV  and  the  projected  future  what-if  states  of  a  UAV.  The  DSV  was  located  at  the 
right  edge  of  each  UAV's  timeline  in  the  Timeline  Display  (Figure  3-1).  For  example, 
the  DSV  in  Figure  3-4  illustrates  that  the  UAV  is  late  to  a  high  priority  target  by  having 
a  dark  gray  rectangle  with  an  H  in  the  rectangle's  center  in  the  late-arrivals  section  of 
the  DSV.  After  the  operator  checks  the  timeline  and  sees  that  the  UAV  is  going  to  be 
late  to  T-7H,  he  highlights  T-7H  on  the  DSV,  posing  a  what-if.  The  DSV  now  shows  the 
projected  future  state  in  the  projection  section.  If  fargef  T-7H's  time  on  farget  is 
delayed,  the  UAV  will  still  be  late  to  a  high  priority  target.  The  light  gray  rectangle 
with  the  H  within  the  rectangle  of  Figure  3-4  illustrates  that  the  UAV  will  still  be  late  to 
a  high  priority  target. 
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Figure  3-4:  Decision  Support  Visualization  (DSV) 

The  green  line  in  Figure  3-5  indicates  the  plotted  and  desired  flight  path. 
Anytime  the  UAV  completely  departed  from  the  green  course  line,  the  UAV  was 
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considered  to  have  deviated.  During  the  simulation,  the  UAV  constantly  fluctuated 
back  and  forth  on  the  flight  path,  but  participants  were  trained  to  respond  when  the 


UAV  actually  was  no  longer  touching  the  course  line. 
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Figure  3-5:  Course-deviation  Illustration. 

Two  tasks  were  selected  in  MAUVE  for  continuous  audio  (sonification) 
representation.  Both  were  secondary  monitoring  tasks,  and  both  were  also  intertwined 
with  the  primary  task  in  a  way  that  the  secondary  task  had  to  be  completed  for  the 
primary  task,  firing  on  the  various  targets  on  the  flight  path,  to  be  successfully 
achieved. 

The  first  secondary  task  was  monitoring  for  late-arrivals  to  targets.  Each  target 
was  either  a  high,  medium,  or  low  priority  target.  On  time  arrivals  and  late-arrivals  to 
high,  medium,  or  low  priority  targets  were  four  discrete  events  to  which  the  continuous 
audio  had  to  be  mapped.  Continuous  audio  could  be  used  to  represent  the  state  of 
UAVs  in  that  the  audio  would  play  continually,  indicating  which  of  the  four  states  a 
UAV  was  in  at  any  point  in  time.  In  particular,  for  this  setup  the  continuous  audio  was 
modulated  so  that  it  would  convey  the  priority  of  the  target  to  which  one  was  late.  For 
example,  as  will  be  discussed  later,  there  were  modulated  increases  in  the  audio  to 
indicate  a  high  priority  target  in  contrast  to  a  low  priority  target.  That  is  what  made 
this  a  sonification. 

Monitoring  for  course-deviations  was  the  other  event  to  which  continuous  audio 
was  mapped.  This  was  an  ideal  continuous  event  in  that  the  UAV  could  start  on  course 
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but  then  on  a  continual  scale  progress  away  from  the  course  line.  Again,  this  changing 
continuous  audio  was  mapped  to  the  rate  of  change  with  which  a  UAV  traveled  off 
course,  making  the  audio  a  sonification. 

3.2.  Four  Auditory-alerts 

Four  auditory  alarms  were  created  to  test  the  effect  of  discrete  and  continuous 
alarms  in  helping  operators  monitor  for  late-arrivals  and  course-deviations  in  MAUVE. 
Presented  in  Figure  3-6,  these  four  auditory- alerts  were  1)  an  oscillating  course- 
deviation  alert,  2)  a  modulated  late-arrival  alert,  3)  a  threshold  course-deviation  alert, 
and  4)  a  threshold  late-arrival  alert. 
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Figure  3-6:  Four  Auditory-alerts. 

The  oscillating  and  modulated  alerts  were  continuous  alarms;  they  provided 
continual  audio  information  about  the  state  of  a  particular  variable  within  the 
simulator.  Instead  of  indicating  a  clear  distinct  threshold,  the  oscillating  and 
modulating  alerts  gradually  increased  as  the  severity  of  either  the  late-arrival  or  course- 
deviation  condition  increased.  The  threshold  alerts  were  designed  to  be  discrete 
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alarms;  they  emitted  a  single  discrete  audio  alert  when  some  precondition  was  met. 
These  four  auditory  alerts  were  presented  to  MAUVE  operators  through  the  HDiSP 
(Headset  Display),  which  will  be  discussed  further  in  the  next  section. 

3.3.  Sensimetrics  HDiSP 

All  audio  alerts  were  presented  equally  in  both  ears  through  the  Sensimetrics 
HDiSPE  Pictured  in  Figure  3-7,  the  HDiSP  is  an  over-the-head  headset  with  sealed 
around-the-ear  ear  cups.  The  HDiSP  provides  ambient  noise  attenuation  ranging  from 
35  to  40  dB  for  the  spectrum  between  500  and  8  kHz.  The  headset  has  integrated  digital 
signal  processors,  which  produce  the  signals  used  in  this  experiment  based  on 
generative  audio  filters  and  processing  of  the  ambient  signal,  which  was  received  from 
microphones  mounted  on  the  headband.  The  next  section  will  provide  the  technical 
description  of  how  each  alert  was  displayed. 


Figure  3-7:  Sensimetrics  Headset  Display  (HDiSP). 

3.4.  Technical  Description  of  Auditory-alerts 

The  auditory-alerts  created  for  this  experiment  were  generated  through 
collaborative  work  between  MIT's  Humans  and  Automation  Laboratory  and  Charles 


*  Contact  Dr.  Thomas  Edward  von  Wiegand,  at  tew@sens.com.  for  further  information  on  the  Sensimetrics  HDiSP. 
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River  Analytics,  Inc.  The  primary  consideration  in  designing  the  audio  alerts  was  to 
create  audio  that  would  capture  the  human  operator's  attention  and  effectively  convey 
the  necessary  information  to  the  operator.  To  design  to  these  parameters,  there  were  a 
few  guidelines  followed  Deatherage  (1972)  and  Sanders  and  McCormick  (1993)). 
Appendix  A  has  a  compilation  of  the  design  principles.  Further,  each  audio  alert  was 
designed  to  always  alert  to  the  worst  condition.  Therefore,  in  the  multiple-UAV 
condition,  where  multiple  UAVs  could  be  having  course  deviations,  the  audio  alert 
played  for  whichever  UAV  currently  was  furthest  off  course. 

The  oscillating  course-deviation  alert  represented  both  the  existence  and  severity 
of  the  UAV  course-deviations.  The  alert  consisted  of  comb  filters  that  were  applied  to  a 
mix  of  pink  noise  and  the  ambient  signal.  Using  pink  noise  means  that  filters  equalized 
the  energy  of  sound  at  each  octave  level,  resulting  in  a  constant  level  presentation  of  the 
audio.  The  mix  ranged  from  0.2  pink  noise  for  low  deviation  to  0.9  pink  noise  for  high 
deviation.  The  comb  filters  had  a  base  delay  of  0.2  ms,  with  a  50  percent  mix  of  the  base 
signal  and  a  feed  forward  delay.  The  delay  values  were  then  oscillated  to  create  a 
periodic  signal.  Because  this  was  a  continuous  audio  scheme,  the  oscillating  course- 
deviation  alert  played  continually  to  provide  an  auditory  image  of  UAV  path  position. 
As  a  UAV  drifted  further  off  course,  the  frequency  of  oscillation  of  the  comb  filter  delay 
decreased  from  17  Hz  to  4.5  Hz,  and  the  depth  of  oscillation  increased  from  0.2  ms  to  0.7 
ms,  on  a  continual  scale. 

The  threshold  course-deviation  alert  consisted  of  a  single  beep  with  a 
fundamental  frequency  of  1000  Hz.  The  beep  lasted  around  8  ms  and  played  whenever 
a  UAV  moved  completely  away  from  the  UAV's  plotted  course  line. 

The  modulated  late-arrival  alert  consisted  of  discrete  harmonic  signals 
continuously  playing  to  indicate  a  projected  late-arrival  at  a  target.  These  sounds 
played  continually  until  the  operator  addressed  the  issue  with  the  corrective  action  of 
choosing  to  delay  the  UAV,  skip  a  target,  or  the  issue  disappeared  because  the  UAV 
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automatically  continued  past  the  target  when  the  UAV  was  going  to  be  late  to  the  next 
target.  The  harmonic  signals  were  composed  of  five  Formant  filters  that  were  applied  to 
a  mix  of  pink  noise  and  the  ambient  signal.  During  the  condition  of  no  late-arrivals,  a 
baseline  audio  condition  was  generated  with  two  filters  set  to  261.6  Hz,  two  filters  set  at 
329.4  Hz,  and  one  filter  set  at  392.0  Hz  (a  major  C  triad  with  extra  balance  in  the  root 
and  third).  If  the  UAV  was  late  to  a  low  priority  target,  a  signal  was  generated  with  two 
filters  set  to  261.6  Hz,  two  filters  set  at  311.1  Hz,  and  one  filter  set  at  392.0  Hz  (a  minor  C 
triad  with  extra  balance  in  the  root  and  third).  If  the  UAV  was  late  to  a  medium  priority 
target,  a  signal  was  generated  with  three  filters  set  to  293.6  Hz  and  two  filters  set  at 
415.3  Hz  (a  tritone  interval  with  a  D  root).  If  the  UAV  was  predicted  to  be  late  to  a  high 
priority  target,  a  signal  was  generated  with  three  filters  set  to  369.9  Hz  and  two  filters 
set  at  523.25  Hz  (a  tritone  interval  with  an  F#  root).  As  the  priority  increased,  the  pink 
noise  mix  also  increased,  from  0.25  for  the  baseline,  to  0.7,  0.8,  and  1.0  for  the  three 
priority  levels. 

Differing  from  the  threshold  course-deviation  alert,  the  threshold  late-arrival 
alert  consisted  of  a  single  beep  with  a  fundamental  frequency  of  415  Hz,  instead  of  1000 
Hz.  In  addition,  the  beep  lasted  around  18  ms,  instead  of  10  ms,  and  played  whenever  a 
UAV  was  projected  to  be  late  to  any  target.  A  UAV  was  determined  late  to  a  target 
when  the  UAV  had  exceeded  the  UAV's  window  of  opportunity  to  destroy  the  target 
and  was  not  going  to  be  able  to  get  over  the  target  in  time  to  fire  on  it  during  the 
allotted  time  window  in  which  the  UAV  was  required  to  be  at  the  target. 
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4.  Methods 


After  integrating  the  audio  alerts  discussed  in  Chapter  3,  Simulator  and  Interface 
Design,  into  MAUVE,  an  experiment  was  conducted  to  compare  continuous  audio  and 
discrete  audio  alerts  in  aiding  single  operator  awareness  in  controlling  multiple 
unmarmed  vehicles.  This  chapter  will  review  the  experimental  questions,  the  setup  of 
the  experiment,  and  the  design  of  the  experiment  for  evaluating  the  research  questions. 

4.1.  Experimental  Questions 

The  objective  of  this  experiment  was  to  determine  whether  continuous  audio 
helps  maximize  the  information  conveyed  to  UAV  operators  more  efficiently  than 
discrete  alarms,  which  are  typically  used  in  current  ground  control  stations.  In 
addition,  the  impact  of  continuous  versus  discrete  alerting  on  operators  moving  from 
control  of  single  to  multiple  UAVs  was  also  a  research  question. 

To  explore  these  objectives,  two  null  hypotheses  were  tested  in  this  research: 

1.  For  various  combinations  of  discrete  and  continuous  alerts,  there  is  no  difference 
in  the  operator's  performance. 

2.  There  is  no  interaction  between  the  discrete  and  continuous  alert  combinations 
and  whether  the  operator  is  controlling  one  or  multiple  UAVs. 

The  alternative  hypothesis  for  the  first  null  hypothesis,  therefore,  was  that  there  is  a 
difference,  and  for  the  second  null  hypothesis,  that  there  is  an  interaction. 
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4.2.  Experimental  Apparatus 

The  experiment  was  administered  in  a  testing  room  with  a  background  ambient 
noise  level  of  ~64dB  on  a  C-scale.  MAUVE  was  run  on  a  multi-modal  workstation 
(MMWS)  (Osga,  VanOrden,  Campbell,  Kellmeyer,  &  Lulue,  2002).  Pictured  in  Figure 
4-1,  the  MMWS  is  a  four-screen  computer  display. 


Figure  4-1:  Multi-Modal  Workstation  (MMWS)  (Osga  et  al.,  2002). 


The  three  screens  across  the  top  were  21  inches  and  set  to  display  at  1280  x  1024 
pixels,  16  bit  color  resolution,  and  the  15  inch  bottom  center  screen  was  set  at  1024  x  768 
pixels,  32  bit  color  resolution.  The  computer  used  to  run  the  simulator  was  a  Dell 
Optiplex  GX280  with  a  Pentium  4  processor  and  an  Appian  Jeronimo  Pro  4-Port 
graphics  card.  Participants  controlled  the  simulator  through  a  generic  corded  computer 
mouse.  Throughout  the  experiment,  the  top  left  display  showed  the  mission  objectives 
and  priorities,  and  the  bottom  center  one  displayed  the  color-coding  for  MAUVE  (Table 
3-1).  The  top  center  display  contained  the  left  MAUVE  map  and  interaction  display,  and 
the  top  right  display  included  the  right  MAUVE  timeline  and  decision  support  display 
(Figure  3-1). 
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For  this  experiment,  the  operator  was  given  the  following  as  the  primary 
objective  for  each  mission:  Make  sure  the  UAV(s)  maximize  the  number  of  targets 
engaged  as  well  as  arrive  back  at  the  base  safely. 

Further,  supervision  of  each  of  the  UAV(s)  for  each  mission  was  broken  down 
into  the  following  prioritized  sub-tasks,  from  highest  priority  to  lowest: 

1.  Return  to  base  (RTB)  within  the  time  limit  for  the  mission  (this  limit  was  clearly 
marked). 

2.  Comply  with  recovery  rules  for  course-deviations. 

3.  Comply  with  recovery  rules  for  target  late-arrivals. 

4.  Destroy  all  targets  before  their  time  on  target  (TOT)  window  ends.. 

5.  Avoid  taking  damage  from  enemies  by  avoiding  all  threat  areas. 

6.  Acknowledge  all  "Push"  radio  calls. 

Participants  were  trained  to  follow  these  priorities,  and  the  objectives  and 
priorities  above  were  displayed  to  them  throughout  the  experiment.  Within  the 
priorities,  recovery  rules  of  how  to  recover  the  UAV  from  a  course-deviation  or  late- 
arrival  were  also  displayed.  The  instructions  on  how  to  comply  with  recovery  rules  for 
course-deviations  were  to  click  "Reset  Navigation"  on  the  control  panel  (Figure  3-2). 
This  theoretically  reset  the  onboard  navigation  in  the  UAV  and  caused  that  UAV  to  fly 
back  to  the  plarmed  flight  path. 

The  instructions  on  how  to  comply  with  recovery  rules  for  target  late-arrivals 
were  as  follows: 

•  Low  Priority  -  Click  "Skip  Target." 

•  Medium  Priority  -  Click  "Skip  Target"  or  employ  the  decision  support 
visualization  (DSV). 

•  High  Priority  -  Employ  DSV  before  requesting  delay  or  clicking  "Skip  Target." 
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For  low  priority  targets,  clicking  the  "Skip  Target"  button  on  the  control  panel  (Figure 
3-2)  would  cause  the  UAV  to  proceed  immediately  to  the  next  target.  For  medium  and 
high  priority  targets,  the  option  was  given  to  use  the  DSV.  As  discussed  in  Chapter  3, 
Simulator  and  Interface  Design,  the  DSV  (Figure  3-4)  is  a  tool  that  illustrates  the  effect  that 
delaying  the  time  on  target  for  one  late  target  has  on  the  remaining  targets  for  that 
UAV's  flight  plan.  With  the  assistance  of  the  information  presented  in  the  DSV, 
participants  then  decided  whether  they  should  click  the  "Skip  Target"  button  or  the 
"Request  Delay"  button  on  the  DSV.  If  they  clicked  "Request  Delay,"  participants  then 
had  to  wait  to  see  if  a  delay  was  granted,  and  if  the  delay  was  granted,  the  UAV  was 
then  no  longer  late  to  the  target  and  would  be  able  to  destroy  the  target.  For  a  high 
priority  target,  the  immediate  response  was  to  employ  the  DSV. 

4.3.  Experimental  Design 

4.3.1.  Independent  Variables 

The  experiment  was  a  4x2  fixed  factor  repeated  measures  model,  with  two 
independent  variables:  the  audio  condition  (a  between-subjects  treatment),  and  the 
number  of  vehicles  under  control  (a  repeated  within-subjects  factor). 

The  four  levels  of  the  audio  condition  factor  were  combinations  of  the  four 
auditory-alerts  described  earlier  in  Figure  3-6  in  Chapter  3,  Simulator  and  Interface 
Design.  Every  participant  was  exposed  to  two  of  the  alerts  shown  in  Figure  3-6. 
Participants  were  presented  one  audio  alert  from  the  course-deviation  columns,  either 
the  oscillating  course-deviation  alert  or  the  threshold  course-deviation  alert,  and  one 
audio  alert  from  the  late-arrival  column,  either  the  modulated  late-arrival  alert  or  the 
threshold  late-arrival  alert.  The  four  audio  conditions  were  the  following:  (1)  the 
threshold  audio  condition  for  both  the  late-arrivals  and  course-deviations  (BothThresh), 
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(2)  the  continuous  oscillating  course-deviation  audio  condition,  with  threshold  alert  for 
the  late-arrivals  (DevCont),  (3)  the  continuous  modulated  late-arrival  audio  condition, 
with  a  threshold  alert  for  course-deviations  (LateCont),  and  (4)  the  both  continuous 
audio  condition,  which  consisted  of  the  oscillating  course-deviation  alert  and  the 
modulated  late-arrival  alert  (BothCont). 

The  second  independent  variable,  the  number  of  vehicles  under  control,  had  two 
levels:  single-UAV  and  multiple-UAV.  In  the  single  level,  the  participant  supervised 
only  one  UAV,  while  in  the  multiple  level,  the  participant  supervised  four  UAVs.  The 
framework  of  this  experiment  is  represented  in  Table  4-1. 

The  experiment  was  counterbalanced;  a  random  half  of  the  participants 
completed  the  single-UAV  scenario  first,  and  the  other  half  finished  the  multiple-UAV 
scenario  first.  Additionally,  the  participants  were  randomly  assigned  to  the  four  audio 
schemes  to  avoid  confounding  effects. 


Audio  Scheme 
(Between) 


Table  4-1:  Experimental  Conditions 

Scenario  (Repeated) 


Single-UAV 

Multiple-UAV 

BothThresh 

Threshold  Course-deviation, 

Threshold  Late-arrival 

Participants  1-9 

Participants  1-9 

DevCont 

Oscillating  Course-deviation, 

Threshold  Late-arrival 

Participants  10-19 

Participants  10-19 

LateCont 

Threshold  Course-deviation, 

Modulated  Late-arrival 

Participants  20-29 

Participants  20-29 

BothCont 

Oscillating  Course-deviation, 

Modulated  Late-arrival 

Participants  30-39 

Participants  30-39 
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4.3.2.  Dependent  Variables 

Dependent  variables  included  the  number  of  missed  course-deviations,  the 
reaction  time  to  correct  course-deviations,  the  number  of  missed  late-arrivals,  the 
reaction  time  to  correct  projected  late-arrivals,  the  number  of  missed  radio  calls,  and  the 
NASA  TLX  scores.  These  are  discussed  in  detail  below. 

4.3.2. 1.  Course  Deviation  Errors  of  Omission 

In  each  test  scenario,  participants  were  expected  to  respond  to  four  triggered 
course-deviations.  See  Appendix  B  for  a  timeline  of  the  events  for  both  the  single  and 
multiple-UAV  scenarios.  Again,  as  illustrated  in  Figure  3-5,  a  course-deviation  was 
defined  as  a  UAV  no  longer  appearing  to  follow  its  planned  flight  path.  Anytime  the 
participant  failed  to  respond  to  one  of  four  triggered  course-deviations,  an  error  of 
omission  was  counted.  This  simply  meant  that  the  participant  failed  to  recognize  and 
address  one  of  the  four  cued  course-deviations  before  the  UAV  self-corrected.  In  the 
event  that  a  course-deviation  was  missed,  this  data  point  was  treated  as  a  missing  data 
point  for  calculating  the  course-deviation  reaction  times. 

4.3.2. 2.  Course-deviation  Reaction  Times 

The  reaction  time  for  a  course-deviation  event  was  the  time  taken  by  the 
participant  to  make  a  corrective  input  after  one  of  the  four  course-deviations  had  been 
triggered.  The  course-deviation  reaction  time  used  for  data  analysis  was  an  average  of 
the  reaction  times  for  the  course-deviations  that  the  participant  responded  to  per  test 
scenario. 

4.3.2. 3.  Late-arrival  Errors  of  Omission 

Four  late-arrivals  occurred  in  each  test  scenario.  Illustrated  in  Figure  3-3,  late- 
arrivals  were  caused  when  a  UAV  slowed  down  because  of  unforeseen  headwinds  and 
was  no  longer  able  to  reach  a  target  in  time  to  complete  the  firing  mission.  Errors  of 
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omission  for  late-arrivals  occurred  when  the  participant  failed  to  respond  to  one  of  the 
four  triggered  late-arrivals.  This  simply  meant  that  the  participant  failed  to  recognize 
and  address  the  late-arrival  before  the  UAV  automatically  moved  past  the  target.  In  the 
event  that  a  late-arrival  was  missed,  this  data  point  was  treated  as  a  missing  data  point 
for  calculating  the  late-arrival  reaction 

4.3.2.4.  Late-arrival  Reaction  Times 

The  late-arrival  reaction  time  was  the  time  taken  by  the  participant  to  make  a 
corrective  action  after  one  of  the  four  late-arrivals  was  indicated.  The  late-arrival 
reaction  time  used  for  data  analysis  was  an  average  of  the  reaction  times  for  the  late- 
arrivals  that  the  participant  responded  to  per  test  scenario. 

4.3.2.5.  Secondary  Workload  Assessment  (Radio  Calls  Missed) 

As  a  secondary  workload  task,  the  number  of  missed  radio  calls  was  an 
indication  of  the  operator's  level  of  mental  workload.  The  count  of  missed  radio  calls 
measured  spare  mental  capacity.  Participants  were  instructed  to  monitor  a  recording  of 
continual  air  traffic  radio  chatter  for  the  word  "Push."  The  word  "Push"  occurred  62 
times  in  a  30  minute  session,  with  an  average  of  27  seconds  between  each  "Push"  radio 
call.  To  acknowledge  the  radio  call,  participants  clicked  an  "Acknowledge  Push" 
button  on  the  display  (Figure  3-2). 

4.3.2. 6.  Subjective  Workload  Assessment  (NASA  TLX  Score) 

The  National  Aeronautics  and  Space  Administration  (NASA)  Task  Load  Index 
(TLX)  gathered  participants'  subjective  assessment  on  a  scale  of  1  to  20  of  mental 
demand,  physical  demand,  temporal  demand,  effort,  performance,  and  frustration.  A 
participant  was  then  asked  to  rate  these  six  dimensions  against  each  other  to  determine 
their  importance  in  the  participant's  workload  (Flart  &  Staveland,  1988).  Since  there  was 
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no  physical  demand  in  this  experiment,  each  participant  was  told  to  rate  the  physical 
demand  so  that  it  was  zeroed  out  of  the  score. 

4.4.  Participants 

Given  an  a  priori  power  analysis  for  a  power  of  0.80,  the  estimated  minimum 
number  of  needed  participants  was  determined  to  be  23,  and  44  were  recruited.  Each 
participant  was  paid  $10  an  hour  to  participate.  The  experiment  took  between  2.5  and  3 
hours  to  complete.  Participants'  ages  ranged  from  20  years  to  42  years,  with  an  average 
age  of  26  years  and  standard  deviation  of  6  years.  There  were  3  Navy  midshipmen,  1 
Army  reservist  specialist,  1  Air  Force  staff  sergeant,  and  39  officers  from  the 
Army/Navy/Air  Force.  Overall,  the  personnel  tested  had  a  combined  experience  of  over 
250  years  of  active  duty  military  service,  with  each  member  having  an  average  of  5.8 
years  of  active  duty  service.  Five  pilots  contributed  as  test  participants,  and  most  of  the 
28  junior  officers  tested  will  be  future  military  pilots.  Overall  there  were  7  female  and 
37  male  participants. 

For  the  data  analysis,  5  participants  were  dropped  because  of  problematic  data. 
The  first  two  were  omitted  because  of  the  failure  of  the  test  proctor  to  administer  proper 
training.  The  third  was  dropped  because  of  a  3  day  interruption  between  the  single- 
UAV  and  multiple-UAV  test  sessions.  The  fourth  participant  was  not  used  because  his 
secondary  task  data  (i.e.,  number  of  push  call  responses)  was  an  outlier  of  more  than 
3.29  standard  deviations  from  the  mean  for  both  the  single-UAV  and  multiple-UAV 
scenarios.  The  fifth  participant  dropped  was  an  active  duty  Air  Force  lieutenant  colonel 
with  4  years  experience  as  a  maintenance  officer  and  an  additional  4  years  of  flight  line 
time  as  a  pilot  and  flight  test  engineer.  He  reported  having  been  diagnosed  with 
tinnitus  and  loss  of  high  frequency  tone  in  one  ear.  During  the  audio  training,  he 
reported  not  being  able  to  hear  the  differentiation  between  a  low  and  high  course- 
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deviation  with  the  oscillating  course-deviation  alert.  His  hearing  test  results  indicated 
age-induced  hearing  loss. 

4.5.  Testing  Procedures 

Each  participant  experienced  three  separate  consecutive  phases  in  completing 
the  experiment.  Each  had  a  60  to  70  minute  training  session,  followed  by  a  70  to  80 
minute  test  session  and  then  a  10  minute  post- test  survey.  Eor  the  training  session,  each 
participant  completed  a  consent  form  and  demographics  survey,  received  standardized 
training  from  a  PowerPoint®  tutorial,  a  hearing  test,  audio  training  for  the  specific 
audio  test  condition,  and  a  thirty  minute  practice  test  session  with  a  trial  run  of  the 
NASA  TLX.  If  required,  a  participant  could  do  an  additional  10  minute  practice  session 
to  ensure  understanding  of  the  proper  MAUVE  control  actions.  Copies  of  the 
participant  consent  form,  demographics  survey,  tutorial,  and  post-test  survey  are  in 
Appendices  C,  D,  E,  and  F,  respectively. 

The  hearing  test  was  designed  to  identify  whether  the  participant  suffered  any 
hearing  loss  (either  temporary  or  permanent).  For  the  hearing  test,  participants  listened 
to  a  CD  of  test  tones  from  40dBSPL  down  to  16dBSPL,  with  5  frequencies  tested  (500 
Hz,  1  kHz,  2  kHz,  4  kHz,  and  8  kHz).  Participants  acknowledged  how  many  beeps 
were  heard  in  each  ear  and  each  frequency.  The  range  tested  ensured  participants  could 
hear  to  within  3dB  of  the  noise  floor  of  the  headset  in  the  normal  lab  conditions.  Based 
on  the  method  of  descending  limits,  the  test  was  designed  so  that  hearing  5  beeps  or 
more  at  each  frequency  showed  adequate  hearing  (Snodgrass,  1975;  Wundt,  1902).  If 
fewer  beeps  were  heard  at  8  kHz  than  at  1  kHz,  the  participant  exhibited  high 
frequency  loss  (Cooper  &  Owen,  1976;  Humes,  Joellenbeck,  &  Durch,  2006).  Age- 
related  permanent  loss  showed  up  most  at  high  frequencies  (i.e.  8  kHz  congestion  and 
other  losses  show  up  at  lower  frequencies  and  across  range)  (Walden,  Prosek,  & 
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Worthington,  1975).  The  hearing  test  was  generated  by  Dr.  Thomas  Edward  von 
Wiegand  of  Sensimetrics  Corporations,  in  conjunction  with  the  HDiSP. 

The  audio  training  gave  each  participant  exposure  to  all  of  the  audio  alerts  each 
participant  would  hear.  Participants  were  presented  with  their  respective  late-arrival 
and  course-deviation  alerts,  as  well  as  the  radio  chatter.  After  introducing  each  of  these 
individually,  they  were  played  to  provide  examples  of  real  test  scenarios.  After  a 
demonstration  of  the  audio  condition,  participants  were  encouraged  to  play  with  the 
audio  demonstration  software  until  they  were  comfortable  with  recognizing  each  of  the 
alerts. 

After  the  training  was  completed,  each  participant  then  completed  both  the 
single-UAV  scenario  and  the  multiple-UAV  scenario.  At  the  conclusion  of  every  test 
session,  MAUVE  generated  a  data  log,  with  all  the  previously  discussed  dependent 
variables.  Additionally,  participants  completed  a  NASA  TLX  following  each  test 
session.  At  the  conclusion  of  the  test  session,  each  participant  completed  a  post-test 
survey,  was  paid,  and  thanked  for  his  time  and  involvement. 

4.6.  Data  Collection 

For  the  collection  of  the  data,  all  pertinent  action  sequences  in  MAUVE,  such  as 
the  reaction  times  to  correct  course-deviations  and  late-arrivals  and  the  radio  call 
acknowledgements  of  the  "Push"  radio  calls,  were  recorded  in  a  MAUVE  data  file  for 
each  scenario.  In  addition,  an  Excel®  file  with  the  NASA  TLX  score  was  generated  and 
saved  for  each  scenario,  and  a  Camtasia®  video  recording  screen  capture  of  the  two 
MAUVE  displays  was  saved. 
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5.  Results 


For  statistical  analysis,  repeated  measures  analysis  of  variances  (ANOVAs)  were 
conducted  for  each  dependent  variable.  The  audio  scheme  factor  was  a  between- 
subjects  factor  for  the  four  audio  conditions,  and  the  scenario  was  a  repeated  within- 
subjects  factor  of  single-UAV  versus  multiple-UAV  levels.  Excluding  the  late-arrival 
reaction  times  and  error  of  omission  counts  which  were  non-parametric,  all  of  the 
remaining  variables  analyzed  met  normality  and  homogeneity  assumptions.  While  a 
was  set  to  0.05,  p  values  between  0.05  and  0.10  were  considered  marginally  significant. 
See  Appendix  G  for  further  details  on  the  statistical  analysis  tests. 

5.1.  Course-deviation  Errors  of  Omission 

Out  of  the  156  course-deviations  presented  to  the  39  participants,  9  course- 
deviations  were  missed  (5.8%).  Of  the  missed  course-deviations,  approximately  half  (4 
course-deviations)  were  missed  by  participants  in  the  LateCont  audio  condition.  The 
data  was  not  normal,  and  the  non-parametric  tests  (Kruskal- Wallis)  showed  no 
significant  differences  between  the  different  audio  conditions. 

5.2.  Course-deviation  Reaction  Times 

Figure  5-1  shows  the  means  for  the  course-deviation  reaction  times  across  the 
four  audio  conditions  (BothThresh,  DevCont,  LateCont,  and  BothCont),  with  standard 
error  bars.  The  omnibus  test  shows  there  was  a  statistically  significant  difference  in  the 
performance  of  participants  based  on  the  audio  scheme  (F(3,35)=2.878,  p=.05),  and  a 
marginally  significant  difference  due  to  the  scenario  (F(l,35)=3.215,  p=.08).  Interaction 
was  not  significant.  After  the  omnibus  testing,  a  Tukey  post-hoc  comparison  showed 
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that  there  was  one  primary  difference  across  the  audio  scheme.  The  significant 
difference  was  between  the  BothCont  and  BothThresh  audio  conditions  for  the  single- 
UAV  scenario  (p=0.02).  There  was  not  a  significant  difference  between  any  of  the  other 
points. 
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Figure  5-1:  Course-deviation  Reaction  Times  Treatment  Means  Plot. 

In  Figure  5-1  there  was  an  apparent  division  in  the  data  based  on  which  audio 
alert  was  present  for  the  course-deviation.  As  part  of  the  post-hoc  analysis  based  on  the 
data  clustering  in  Figure  5-1,  with  the  BothCont  and  DevCont  appearing  in  one  group 
and  the  LateCont  and  BothThresh  in  another  group,  the  model  was  reconfigured  as  a 
2x2  fixed  factor  repeated  measures  model  with  audio  scheme  and  scenario  as  the 
independent  variables,  respectively  (Table  5-1).  Instead  of  the  four  audio  conditions, 
the  factor  levels  were  collapsed  into  two  levels:  those  participants  exposed  to  the 
continuous  oscillating  course-deviation  alert  (participants  in  the  BothCont  or  DevCont 
audio  conditions)  and  those  exposed  to  the  threshold  course-deviation  alert 
(participants  in  the  LateCont  and  BothThresh  audio  conditions).  The  audio  scheme 
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remained  a  between-subjects  factor,  while  the  scenario  remained  a  repeated  within- 
subjects  factor. 


Table  5-1:  Post  Hoc  Experimental  Conditions. 


Scenario  (Repeated) 


Audio 

Scheme 

(Between) 


Single-UAV 

Multiple-UAV 

Continuous  Oscillating 

Course-deviation  Alert 

(BothCont  or  DevCont) 

Participants  1-20 

Participants  1-20 

Threshold 

Course-deviation  Alert 

(LateCont  or  BothThresh) 

Participants  21-39 

Participants  21-39 

As  seen  in  Figure  5-2,  the  analysis  confirms  that  the  participants  with  the 
continuous  oscillating  alert  condition  performed  significantly  better  than  those  with  the 
threshold  course-deviation  alert  condition  (F(l,3'7)  =  8.874,  p=.01). 


»  Continuous  Oscillating  Course-Deviation  Alert 
—  ■ —  Threshold  Course-Deviation  Alert 


Figure  5-2:  Post  Hoc  Analysis  Course-deviation  Reaction  Times  Treatment  Means  Plot. 
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5.3.  Late-arrival  Errors  of  Omission 


All  39  participants  responded  to  all  of  the  4  late-arrivals  presented  to  each  of 
them,  thus  there  were  no  late-arrival  errors  of  omission. 

5.4.  Late-arrival  Reaction  Times 

The  original  late-arrival  reaction  times  were  not  normally  distributed,  thus  a 
natural  log  transformation  was  performed.  The  omnibus  results  showed  a  significant 
difference  in  the  performance  of  participants  between  the  four  audio  conditions 
(F(3,35)=3.345,  p=.03)  and  a  significant  difference  due  to  the  scenario  (F(l,35)=20.737, 
p<.001).  Interaction  was  not  significant. 

As  seen  in  Figure  5-3,  there  was  an  apparent  split  in  the  data.  A  post-hoc  Tukey 
comparison  revealed  there  was  a  significant  or  marginally  significant  difference 
between  the  data  for  the  DevCont  audio  condition  and  the  other  three  audio  conditions: 
BothThresh  audio  condition  (p=.05),  LateCont  audio  condition  (p=.08),  and  BothCont 
audio  condition  (p=.07).  These  post-hoc  comparisons  confirmed  the  apparent 
differences  within  the  treatment  means  plot  in  Figure  5-3. 

5.5.  Workload 

To  measure  workload,  the  performance  on  a  secondary  task  was  monitored,  and 
scores  for  subjective  workload  were  recorded.  These  are  detailed  in  the  following 
sections. 

5.5.1.  Secondary  Task  Assessment 

For  the  number  of  missed  radio  calls,  there  was  no  significant  difference  between 
the  audio  conditions  or  across  the  scenarios.  According  to  this  secondary  workload 
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»  BothCont  — ■ —  LateCont  — ^  -  DevCont  -  -  BothThresh 

Figure  5-3:  Transformed  (Natural  Log)  Late-arrival  Reaction  Times  Treatment  Means  Plot, 
measurement,  the  participants  were  equally  saturated,  regardless  of  their  assigned 
audio  condition  or  whether  they  supervised  a  single  or  multiple  UAVs.  This  was  not  a 
surprise;  as  discussed  before  and  illustrated  in  Appendix  B,  the  scenarios  were  designed 
with  an  equal  number  of  tasks. 

5.5.2.  Subjective  Assessment 

Similar  to  the  secondary  workload  measurement,  the  subjective  measurement  of 
the  NASA  TLX  scores  was  not  significantly  different  across  audio  conditions  or 
scenarios.  The  non-significant  results  across  the  scenario  also  were  not  a  surprise, 
because  as  illustrated  in  Appendix  B,  the  single-UAV  and  multiple-UAV  scenarios  were 
designed  to  have  the  same  number  and  placement  of  events.  In  the  multiple-UAV 
scenario,  the  number  of  events  and  same  point  in  time  of  the  events  was  just  split  over 
four  UAVs  instead  of  all  being  on  one  UAV's  flight  path. 
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5.6.  Post-Experiment  Subjective  Responses 

Subjective  responses  regarding  the  usefulness  of  the  audio  cues,  whether  discrete 
or  continuous,  were  overwhelmingly  positive.  In  reviewing  the  participants'  comments, 
some  valuable  insights  were  gained. 

The  overarching  feeling  toward  the  discrete  threshold  alerts  was  that  while  they 
were  better  than  nothing,  the  discrete  threshold  alerts  could  have  been  improved. 
Many  felt  that  these  audio  alerts  were  nice  to  have  as  they  "made  [the  participant]  pay 
attention"  and  were  "a  good  back-up  for  the  visual  cue."  The  majority  commented  the 
audio  alerts  worked  well  in  conjunction  with  the  visual  display  to  better  enhance  their 
awareness.  One  participant  reported  the  audio  alert  "was  especially  helpful  because 
noticing  course-deviations  visually  was  difficult." 

A  key  concern  raised  about  the  threshold  audio  alerts  was  that  the  alerts  were 
"hard  to  distinguish  from  background  noise."  The  feeling  was  that  the  beeps  blended 
into  the  background  too  easily.  Threshold  audio  alerts  were  masked  by  things  such  as 
the  general  radio  chatter  or  other  alerts  in  the  mission  such  as  the  beep  that  alerted  the 
operator  of  new  threats  to  avoid.  One  reported  he  found  the  threshold  audio  alert  to  be 
"too  short  and  too  discrete  to  call  my  attention.  I  usually  noticed  the  late-arrivals 
visually."  Others  offered  possible  solutions.  One  said  the  audio  alert  "should  be  more 
annoying  and  should  last  longer."  A  common  feeling  was  that  if  the  audio  alert  was 
"longer  and  more  prominent,"  the  audio  would  be  more  effective  as  a  supplemental 
tool  to  the  visual  alert.  In  fact,  over  half  of  the  participants  exposed  to  one  of  the 
discrete  alerts  asked  for  alerts  with  "more  than  just  a  simple  beep"  and  wished  for  a 
more  repetitive  audio  alert  that  would  "last,"  so  operators  would  have  to  address  the 
alert. 

Another  concern  raised  in  the  threshold  audio  alert  feedback  was  that  the  audio 
alerts  were  "difficult  to  use  effectively  with  four  UAVs."  Some  suggested,  "It  might 
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have  been  more  helpful  if  it  referred  to  the  selected  UAV."  The  results  of  this  research 
support  this  observation.  Participants  needed  audio  support  that  would  point  them  to 
the  problem  UAV,  not  just  the  fact  that  there  was  a  problem  with  one  of  the  UAVs. 

The  continuous  audio  alerts  were  generally  appreciated  as  situational  enhancers 
and  attention  managers,  but  some  felt  the  continuous  audio  was  potentially  fatiguing. 
Although  generally  positive,  participants  did  report  the  continuous  modulated  late- 
arrival  alert  "focused  attention  to  late  UAVs."  Like  the  discrete  threshold  alerts, 
continuous  audio  was  "audio  backup"  to  the  visual  display  and  "alerted  [participants] 
to  the  impending  late-arrivals  more  quickly."  With  continuous  audio,  participants 
learned  to  use  the  continuous  nature  to  supplement  their  regular  visual  scan,  with  one 
recounting,  "I  mostly  looked  at  the  left  screen"  and  "the  audio  was  the  cue  to  look  at  the 
right  screen."  Though  one  reported  that  the  continuous  modulated  late-arrival  alert 
"helped  with  many  UAVs"  and  another  that  it  was  a  "good  alert  of  a  late-arrival," 
participants  found  it  difficult  to  distinguish  the  priority  of  the  target  based  on  the  audio 
alone. 

While  the  modulated  late-arrival  alert  was  touted  as  an  enhancement  tool,  it  was 
noted  by  participants  that  the  tones  "provoked  a  stress  response  so  that  [the  participant] 
felt  more  anxious."  An  Air  Force  Predator  pilot  noted  that  the  modulated  late-arrival 
alert,  "while  effective  at  alerting  to  issues,  was  extremely  armoying,"  and  he  "would  not 
subject  [himself]  to  that  background  noise  for  hours  at  a  time  day  after  day." 

Similar  to  the  modulated  late-arrival  alert,  the  continuous  oscillating  course- 
deviation  alert  was  lauded  as  "useful  as  a  backup  secondary  cue."  A  negative  comment 
about  this  continuous  audio  alert  was  that  the  audio  alert  could  "contribute  greatly  to 
fatigue"  over  a  long  mission.  Ten  percent  of  the  participants  exposed  to  the  continuous 
oscillating  course-deviation  suggested  that  this  additional  alert  be  left  off  throughout 
the  bulk  of  the  mission  and  only  have  the  additional  alert  on  during  particular  parts  of 
the  mission.  This  way  the  effectiveness  of  the  tool  could  be  used  when  the  operator 
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needed  an  additional  cue.  At  the  same  time,  operators  could  be  spared  the  annoyance 
of  continuous  audio  input  over  a  long  duration  mission,  when  they  were  not 
continually  at  a  peak  stress  and  work  level. 

Interestingly,  all  of  the  negative  comments  regarding  the  oscillating  course- 
deviation  alert  came  from  the  participants  who  were  exposed  to  both  the  oscillating 
course-deviation  alert  and  the  modulated  late-arrival  alert  (BothCont),  instead  of  the 
oscillating  course-deviation  alert  and  the  threshold  late-arrival  alert  (DevCont).  Twenty 
percent  of  those  exposed  to  both  were  annoyed  by  the  oscillating  alert.  Regardless,  the 
majority  felt  the  continuous  audio  alerts  helped,  but  some  time  may  be  needed  for 
operators  to  get  used  to  them.  As  one  operator  stated,  "Effective  as  they  were,  I  just 
couldn't  remember  which  was  which." 

5.7.  Summary  of  Experimental  Findings 

Key  findings  of  the  data  analysis  were  found  in  the  post-hoc  analyses  discussed 
previously  in  this  section.  When  the  course-deviation  reaction  times  were  collapsed, 
those  with  the  continuous  oscillating  alert  condition  were  shown  to  have  performed 
significantly  better  than  those  with  the  threshold  course-deviation  alert  condition. 
Further,  though  the  count  of  error  of  omissions  for  the  course-deviations  was  not 
significant,  it  should  be  noted  that  almost  half  of  all  of  the  missed  course-deviations 
occurred  under  the  LateCont  (comprised  of  threshold  course-deviation  and  modulated 
late-arrival  alerts)  condition. 

Another  key  finding  was  that  for  the  late-arrival  reaction  times,  the  only 
condition  that  had  a  significant  difference  in  performance  from  the  other  conditions  was 
DevCont  (comprised  of  oscillating  course-deviation  and  threshold  late-arrival  alerts). 
The  DevCont  condition  was  shown  to  have  significantly  poorer  performance  than  the 
three  other  audio  conditions:  BothCont  (comprised  of  oscillating  course-deviation  and 
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modulated  late-arrival  alerts),  LateCont  (comprised  of  threshold  course-deviation  and 
modulated  late-arrival  alerts),  and  BothThresh  (comprised  of  threshold  course- 
deviation  and  threshold  late-arrival  alerts). 

The  subjective  feedback  results  showed  an  overall  preference  for  audio  aids  to 
supplement  visual  events,  but  there  was  annoyance  reported  for  some  of  the  continuous 
audio  presentations.  Many  felt  it  would  be  fatiguing  to  run  this  form  of  audio 
continually  throughout  the  long  missions  that  some  UAV  operators  supervise.  Overall 
though,  there  was  a  request  for  more  than  a  discrete,  single  beep  alert. 
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6.  Discussion 


This  chapter  discusses  the  findings  of  the  results  presented  in  Chapter  5,  Results. 
The  discussion  that  follows  presents  these  findings  in  relation  to  multiple  resource 
theory,  mapping  of  audio  alerts,  change  blindness,  and  masking.  It  then  closes  with  an 
explanation  of  the  causes  of  the  workload  results. 

6.1.  Multiple  Resource  Theory 

The  anesthesiology  studies  discussed  in  Chapter  2  showed  that  sonifications 
successfully  supplemented  the  visual  modality  (Watson  &  Sanderson,  2004).  They  did 
not  replace  the  visual  representation  on  the  screen,  but  they  enhanced  the  visual 
display.  Similarly,  within  this  experiment,  as  operators  were  performing  tasks  or  even 
monitoring  the  course-deviation  visually,  operators  relied  on  the  aural  channel  to 
correct  course-deviations  quickly,  which  was  reported  in  their  post-test  subjective 
feedback.  This  illustrates  that  participants  were  integrating  the  audio  input  to  confirm 
and  guide  their  visual  scan  to  more  readily  identify  problems. 

The  multiple-UAV  scenario's  reaction  times  were  expected  to  be  longer  than  the 
single-UAV  scenario's;  however,  there  appeared  to  be  almost  no  differences  in  course- 
deviation  reaction  times  between  the  single-UAV  and  multiple-UAV  scenarios  when 
the  discrete  threshold  course-deviation  audio  alert  was  used.  In  comparison,  the 
scenarios  using  the  continuous  oscillating  course-deviation  audio  alert  showed  the 
expected  trend  of  the  multiple-UAV  scenario  reaction  times  being  longer  than  the 
single-UAV  scenario.  This  difference,  between  the  discrete  and  continuous  audio 
conditions,  is  likely  a  result  of  the  fact  that  in  the  discrete  audio  condition,  operators 
had  already  determined  the  problem  visually  and  then  waited  for  aural  confirmation 
before  addressing  the  problem.  Conversely,  in  the  continuous  audio  condition,  the 
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audio  first  and  more  readily  alerted  the  operator  that  there  was  a  problem.  The 
operator  then  addressed  the  problem  by  visually  determining  the  UAV  that  the 
problem  had  occurred  on,  causing  the  longer  reaction  time  for  the  multiple-UAV 
scenarios. 

The  continuous  audio  provided  constant  auditory  input  that  participants  relied 
upon  to  more  quickly  confirm  their  interpretation  of  the  visual  displays.  Those  exposed 
to  the  discrete  audio  had  to  wait  until  the  threshold  at  which  the  audio  was 
programmed  to  alert  operators  was  met,  before  receiving  any  auditory  confirmation  of 
what  they  may  have  already  perceived  visually.  The  results  of  this  experiment,  for  the 
course  deviation  reaction  times,  illustrate  that  operators  had  the  resources  to  integrate 
the  continuous  audio  with  their  visual  perception,  allowing  them  to  out-perform  those 
with  the  discrete  audio.  Thus,  continuous  audio  better  utilized  the  audio  charmel's  pool 
of  resources. 

6.2.  Mapping  Audio  Alerts  to  Intuitive  Triggers 

While  this  experiment  showed  the  continuous  audio  helped  in  monitoring  course 
deviations,  it  also  demonstrated  a  case  where  continuous  audio  did  not  aid  an  operator 
in  a  supervisory  role.  For  the  late-arrival  reaction  times,  there  was  no  significant  benefit 
gained  by  having  a  continuous  audio  alert  rather  than  a  discrete  audio  alert  for  late 
arrivals.  The  primary  reason  there  was  no  performance  enhancement  with  the 
continuous  modulated  late-arrival  alert  was  that  the  late-arrivals  were  discrete  events  (a 
UAV  was  either  late  or  not  late).  In  relation  to  discrete  audio,  the  continuous  audio  did 
not  provide  any  extra,  useful  information  in  the  late  arrival  condition.  The  audio 
represented  one  of  four  discrete  states,  rather  than  a  continuous  parameter,  so  it  could 
not  convey  progression  towards  a  new  state,  and  thus  the  benefit  of  continuous  audio 
was  not  fully  exploited. 


64 


Conversely,  the  positive  results  of  the  continuous  oscillating  course-deviation 
alert  were  attributable  to  its  intuitive  mapping  to  the  continually  changing  state  of  the 
UAV  course  deviation.  As  the  UAV  moved  off  course,  the  continuous  audio 
progressively  changed,  and  the  participant  was  then  given  useful  information  in  the 
sonification  that  allowed  him  to  choose  his  own  threshold  for  course  deviations,  as 
discussed  above  in  the  multiple  resource  theory  discussion. 

6.3.  Change  Blindness 

In  Chapter  2,  Background,  the  point  was  raised  that  some  forms  of  audio  have 
been  shown  to  help  mitigate  visual  change  blindness  that  may  result  from  relying  on 
only  a  visual  presentation  (Nehme  &  Cummings,  2006).  Because  there  was  no  baseline 
condition,  with  only  visual  displays  and  no  audio  alerts,  there  is  no  empirical  evidence 
that  the  discrete  or  continuous  audio  displays  helped  mitigate  visual  change  blindness. 
It  is  noteworthy,  however,  that  no  late-arrivals  and  an  insignificant  number  of  course- 
deviations  were  missed  by  all  the  participants.  The  indication  is  that  the  combination  of 
the  visual  and  audio  displays  did,  in  fact,  alert  the  participants  to  problem  situations. 

In  terms  of  error  rates  in  responding  to  both  course-deviation  and  late-arrival 
events,  only  a  very  small  percentage  of  course-deviation  alerts  were  missed,  and  all  of 
the  late-arrivals  were  recognized  by  the  participants.  This  slight  difference  was  most 
likely  caused  by  the  visual  alert  representation  in  that  the  late-arrival  visual  alert  was 
easily  seen  by  the  physical  displacement  and  color  change  of  the  target  icon  (Figure  3-3). 
In  contrast,  whether  a  UAV  was  truly  off  course  was  more  visually  ambiguous,  as  is 
seen  in  Figure  3-5.  This  was  exacerbated  by  the  simulation,  which  caused  the  UAV  to 
automatically  resume  its  correct  course  if  a  participant  did  not  recognize  and  correct  an 
off-course  error  within  thirty  seconds.  This  illustrates  the  importance  of  dual  coding 


65 


alerts  on  the  visual  and  aural  channels,  which  is  one  way  to  combat  change  blindness 
and  was  generally  effective  in  this  experiment. 

6.4.  Masking 

The  subjective  feedback  of  the  participants  suggested  that  they  felt  the  discrete 
threshold  alerts  were  masked  by  other  clutter  and  noise  in  the  radio  chatter,  which  is 
why  the  discrete  threshold  alerts  may  not  have  performed  as  well  as  the  continuous 
ones.  In  general,  the  results  showed  that  almost  half  of  the  missed  course-deviations 
occurred  in  the  audio  condition,  where  the  continuous  modulated  late-arrival  audio 
alert  was  present  in  a  scenario  with  the  discrete  threshold  course-deviation  audio  alert. 
Further,  the  results  showed  that  late-arrival  recognition  was  no  different  across 
conditions,  except  for  a  significant  decrease  in  performance  when  the  continuous 
oscillating  course-deviation  alert  was  present  along  with  the  discrete  threshold  late- 
arrival  audio  alert.  Both  of  these  statistical  results  are  examples  of  how  masking  can 
result  in  degraded  performance.  Therefore,  a  continuous  audio  alert  should  be 
integrated  with  the  other  alerts  so  that  masking  of  less  prominent  discrete  or  even 
continuous  alerts  does  not  occur. 

6.5.  Workload 

The  workload  measurements  (subjective  responses  and  secondary  task 
performance)  showed  no  significant  difference  between  the  single-UAV  and  multiple- 
UAV  scenarios.  The  operators  reportedly  perceived  no  difference  in  the  workload, 
which  was  not  a  surprise.  As  discussed  before  and  presented  in  Appendix  B,  the  two 
scenarios  were  designed  with  the  same  number  and  types  of  events.  The  only 
difference  was  that  for  the  multiple-UAV  scenario,  the  events  were  separated  over  four 
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UAVs  instead  of  one  UAV.  The  intent  was  to  provide  the  same  number  of  tasks,  and 
thus  the  same  amount  of  work,  in  each  scenario.  Because  there  was  an  identical  set  of 
events  for  much  of  the  experiment,  participants  likely  recognized  no  difference  in  their 
efforts  between  the  two  scenarios.  This  was  supported  by  the  similar  performance  on 
the  secondary  task  for  the  single-UAV  and  multiple-UAV  scenarios. 

Although  the  workload  measurements  were  not  significant,  the  performance 
metrics  were  different  between  the  scenarios.  The  late-arrival  reaction  times  were 
significantly  different,  and  the  course-deviation  reaction  times  were  marginally 
different.  In  both  cases,  there  were  shorter  reaction  times  for  the  single-UAV  scenario 
than  the  multiple-UAV  scenario.  This  performance  difference  is  likely  the  result  of 
participants  having  to  divide  their  attention  while  monitoring  four  UAVs  for  the 
multiple-UAV  scenario.  The  impact  of  divided  attention  for  multiple  UAV  control  will 
be  discussed  further  in  the  findings  section  of  Chapter  7,  Conclusion. 
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7.  Conclusion 


In  this  concluding  chapter,  the  primary  research  questions  are  answered, 
followed  by  a  discussion  of  workarounds  to  potential  integration  issues  and  areas  for 
future  research. 

7.1.  Findings 

The  primary  questions  addressed  through  this  research  are  as  follows: 

1.  When  compared  with  discrete  audio,  does  continuous  audio  better  aid  human 
supervision  of  UAV  operations? 

2.  Is  the  effectiveness  of  the  discrete  or  continuous  audio  support  dependent  on 
operator  workload? 

7.1.1.  Value  Added  by  Continuous  Audio 

From  the  experiment  results  for  the  single  and  multiple-UAV  scenarios,  it 
appears  that  the  continuous  oscillating  course-deviation  audio  alert  helped  participants 
respond  more  quickly  to  the  task  of  recovering  from  UAV  course-deviations.  The 
continuous  modulated  late-arrival  audio  alert,  in  contrast,  did  not  help  or  hinder  the 
participants  in  responding  to  late-arrivals,  primarily  due  to  a  confound  in  alert 
mapping. 

These  results  highlight  the  importance  of  context  in  alert  design.  When  the 
continuous  oscillating  course-deviation  audio  alert  was  used  with  the  discrete  threshold 
late-arrival  audio  alert,  reaction  times  to  addressing  late-arrivals  were  significantly 
worse  than  all  other  cases,  due  to  masking.  In  contrast,  when  used  with  the  continuous 
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modulated  late-arrival  audio  alert,  the  late-arrival  reaction  times  did  not  suffer,  and  the 
course-deviation  reaction  times  improved. 

Thus,  continuous  audio  alerts  helped  in  UAV  control,  but  they  must  be  added  in 
light  of  all  the  other  audio  input  occurring  in  the  display.  Caution  must  be  taken  so  that 
the  added  benefit  of  a  continuous  audio  alert  is  not  lost  because  of  a  lack  of  proper 
system  integration.  Further,  continuous  audio  must  only  be  applied  in  cases  where  it 
maps  to  the  occurrences  of  the  event  that  is  being  monitored.  If  it  is  a  discrete  event, 
like  late-arrivals,  there  may  not  be  a  gain  in  using  continuous  audio  in  place  of  discrete 
audio. 

7.1.2.  The  Impact  of  Workload 

The  results  indicate  a  decrease  in  performance  when  supervising  multiple  UAVs 
instead  of  a  single  UAV,  regardless  of  the  audio  condition.  The  performance 
degradation  in  the  multiple-UAV  condition  is  not  a  surprise.  Task  load  stress  means 
that  even  with  a  constant  signal  rate,  performance  will  decline  if  more  information 
inflow  must  be  used  (Conrad,  1985).  This  means  that  even  with  the  same  number  of 
tasks,  if  the  tasks  are  distributed  over  multiple  vehicles,  the  performance  will  decrease. 
The  effects  of  load  stress  have  been  illustrated  in  air  traffic  control  (Cummings  & 
Tsonis,  2005)  and  UAV  studies  (Cummings  &  Mitchell,  2008;  Cummings,  Nehme,  & 
Crandall,  2007;  Dixon  et  al.,  2005).  Given  the  results  of  previous  research,  there  is  no 
reason  to  assume  the  results  of  this  study  would  have  been  any  different.  However,  the 
use  of  continuous  audio  displays  was  meant  to  alleviate  some  of  this  increased 
workload,  and  participants  recognized  this  in  the  subjective  responses,  noting  that 
though  the  audio  alerts  were  "very  helpful"  when  focusing  on  other  tasks,  it  was 
"harder  to  comprehend  with  four  UAVs  than  one." 

On  the  whole,  though,  participants  exposed  to  continuous  audio  outperformed 
participants  exposed  to  discrete  audio  in  both  the  single-UAV  and  multiple-UAV 
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scenarios.  Previous  research  controlling  two  UAVs  showed  that  an  audio  alert 
improves  performance  over  a  baseline  condition  of  no  audio  but  that  generally, 
performance  degrades  when  the  number  of  UAVs  under  control  increases  (Dixon  et  al., 
2005).  The  results  of  this  thesis  mirror  this  previous  research,  except  that  it  was  shown 
that  continuous  audio,  when  used  appropriately,  can  actually  mitigate  the  negative 
impacts  of  increased  workload  due  to  increased  numbers  of  UAVs  under  control. 

The  conclusion  to  these  research  questions  is  that  with  correct  application, 
continuous  audio  is  more  helpful  than  discrete  audio  in  supporting  the  supervision  of 
multiple  tasks  over  multiple  vehicles.  While  continuous  audio  may  be  a  performance 
enhancer,  it  is  also  important  to  assess  the  research  and  development,  acquisition,  and 
maintenance  costs  associated  with  fielding  this  new  headphone  technology. 
Quantification  of  these  various  factors  will  determine  whether  such  technology  should 
ultimately  be  employed,  which  will  be  discussed  next. 

7.2.  Integration  Issues 

A  shortcoming  in  the  use  of  headphones  for  the  presentation  of  the  continuous 
audio  is  the  isolation  of  the  operator  from  outside  communication  (Tannen,  1998).  In 
operational  integration  with  military  personnel,  operators  will  often  only  wear  one  half 
of  the  headset  because  wearing  the  headset  on  both  ears  isolates  them  from  inter-team 
interaction.  It  is  important  to  make  sure  that  any  integration  of  a  headset  does  not 
interfere  with  critical  work  environment  constraints,  especially  those  in  a  team  setting. 

Furthermore,  though  continuous  audio  can  promote  objective  performance, 
some  participants  in  the  experiment  noted  an  annoyance  with  long  term  exposure  to 
these  forms  of  continuous  audio.  One  solution  in  integration  is  to  limit  exposure  to  the 
audio  and  only  use  it  when  needed.  For  example,  rather  than  play  continuous  audio 
throughout  an  entire  four-hour  mission,  the  continuous  audio  tool  could  be  active  only 


71 


during  the  portion  of  the  mission  when  workload  is  heavy,  or  when  one  of  the 
peripheral  tasks  reaches  a  cautionary  state  that  may  require  the  human  operator's 
intervention.  Thus,  this  kind  of  adaptive  display  could  either  add  or  remove  audio  aids 
as  required  by  the  workload  situation. 

7.3.  Cost-Benefit  Analysis 

While  the  results  show  that  continuous  audio  displays  improved  operators' 
performance,  the  question  is  to  what  extent?  In  addition,  in  comparison  to  this  benefit, 
what  is  the  cost  to  the  DOD  to  actually  acquire  these  displays? 

To  consider  quantitative  benefits  of  the  continuous  audio  displays,  the  reaction 
times  to  course-deviations  and  late-arrivals  are  reviewed.  The  results  show  that  with 
the  continuous  audio,  participants  were  on  average  2  seconds  or  6  seconds  faster  at 
responding  to  course-deviations  or  late-arrivals,  respectively,  than  the  participants  with 
the  discrete  audio  display.  Thus,  these  experimental  results  show  a  performance 
enhancement  of  a  31  percent  decrease  in  reaction  times  for  monitoring  tasks  with  the 
aid  of  continuous  audio  displays  instead  of  the  aid  of  discrete  audio  displays.  In 
aviation  this  time  interval  can  be  critical,  particularly  in  UAS  operations  where 
operators  deal  with  time  lags  in  the  control  of  remote  vehicles.  Linking  continuous 
audio  to  support  operators  in  monitoring  time  critical  events  could  be  very  beneficial. 

In  terms  of  the  actual  hardware,  the  HDiSP  is  still  a  new  technology  and  is  the 
only  headset  of  its  kind.  The  HDiSP  used  in  this  experiment  was  a  prototype  version 
provided  by  Sensimetrics.  To  date,  Sensimetrics  is  only  providing  HDiSP  for  custom 
orders.  Two  HDiSPs,  along  with  the  software,  have  been  sold  at  a  cost  of  3,750  dollars 
per  headset  (T.  E.  von  Wiegand,  personal  communication,  April  2,  2008).  Sensimetrics 
estimates  that  if  they  were  mass  producing  the  HDiSP,  the  headset  would  sell  between 
1,000  and  2,000  dollars  per  headset  (T.  E.  von  Wiegand,  personal  communication,  April 
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2,  2008).  In  contrast,  the  best  aviation  headsets  cost  about  300  dollars  (Sky Geek,  2008). 
Overall  the  price  increase  of  purchasing  the  HDiSP  instead  of  aviation  headsets  would 
be  about  1,200  dollars  (400  percent),  if  the  HDiSP  was  mass  produced  at  1,500  dollars 
per  headset. 

Rough  performance  and  cost  increases  can  be  calculated,  but  these  numbers 
mean  nothing  unless  placed  in  some  context.  For  instance,  in  the  motivation  section  of 
Chapter  1,  Introduction,  the  statistic  was  cited  that  over  the  next  8  years,  15.7  billion 
dollars  will  be  spent  on  UAVs  in  the  United  States,  yielding  11,000  UAVs  in  operation 
(Tsach  et  al.,  2007).  Equipping  each  UAV  with  an  HDiSP  for  single  operator 
supervision  would  result  in  a  cost  of  13  million  dollars,  which  is  less  than  0.1  percent  of 
all  the  money  projected  to  be  spent  on  UAVs  in  the  United  States  in  the  next  8  years. 

While  there  is  a  cost  to  integrating  a  new  continuous  audio  system,  there  are  also 
research  and  development  costs  for  continuous  audio  displays  and  HDiSP  technology. 
Further,  there  would  likely  be  additional  maintenance  costs  for  the  HDiSP,  which  is  a 
more  complex  hardware  device  that  may  not  be  as  sturdy  in  field  conditions  as  older, 
more  robust  aviation  headphones.  Dependent  upon  implementation,  if  there  were 
significant  maintenance  issues,  there  would  be  social  costs  as  well.  Operators  might 
lose  trust  or  grow  frustrated  with  the  constant  maintenance  and  then  stop  using  the 
device. 

The  results  of  the  experiment  in  this  thesis  illustrate  continuous  audio  displays 
provide  a  performance  enhancement.  When  compared  with  participants'  performance 
with  discrete  audio  displays,  the  continuous  audio  displays  are  shown  to  decrease 
reaction  times  by  31  percent.  An  initial  cost  benefit  analysis  shows  that  the  cost  of 
implementing  such  a  device  is  minimal,  but  future  research  is  needed  to  do  a  full 
analysis  of  economic,  operational,  and  social  costs  of  implementing  these  new  audio 
displays  into  UAS  interfaces. 
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7.4.  Limitations  and  Future  Work 


•  A  limitation  of  this  research  is  that  it  assumes  a  certain  level  of  autonomy  in  the 
unmarmed  vehicles,  such  as  the  vehicles  flying  themselves  from  waypoint  to 
waypoint,  with  the  operator  performing  a  distinct  payload  mission.  The  research 
presented  here  focuses  on  the  human  operator  acting  in  a  higher  supervisory 
role. 

•  The  tasks  represented  by  the  continuous  audio  alerts  were  not  primary  tasks,  but 
secondary  tasks  that  required  occasional  operator  input.  The  results  may  have 
been  different  had  the  continuous  audio  alerts  been  linked  directly  to  the 
primary  task. 

•  Within  the  MAUVE  simulator,  there  is  a  rapid  onset  of  change  for  the  course- 
deviation  and  late-arrival  events.  Future  research  could  be  done  on  a  simulator 
that  allows  for  a  more  gradual  onset  of  change  to  further  explore  how  the  rate  of 
increase  for  the  audio  intensity  affects  the  speed  and  confidence  of  response  by 
the  participants.  In  particular,  how  the  onset  of  change  for  the  continuous  audio 
helps  alert  the  participant  to  the  problem  is  an  area  of  future  research. 

•  While  continuous  audio  alerts  have  been  proven  beneficial  in  comparison  to 
discrete  audio  alerts,  further  research  could  be  completed  to  investigate  the 
different  patterns  of  sonifications  to  see  which  is  best  for  helping  operators 
supervise  multiple  tasks  on  multiple  vehicles. 

•  Another  future  study  could  compare  the  benefits  of  continuous  audio  alerts  with 
spatial  audio  and  test  the  integration  of  continuous  audio  into  a  spatial  audio 
presentation.  The  point  of  this  future  study  would  be  to  test  the  value  added 
when  continuous  audio  is  used  with  or  in  addition  to  spatial  audio. 

•  These  results  suggest  that  perhaps  a  better  implementation  of  continuous  audio 
alerts  is  to  use  them  only  during  high  workload  situations,  so  as  to  minimize  any 
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annoyance  factor,  while  maximizing  the  objective  benefits  of  the  tool.  Future 
research  could  test  ways  to  build  the  sonifications  into  adaptive  audio  displays 
that  change  the  amount  and  type  of  audio  output  based  on  the  workload 
presented  by  the  system  to  the  operator. 

•  Another  extension  of  this  research  would  be  investigating  how  haptic  cueing, 
either  as  a  replacement  or  an  addition  to  the  continuous  audio  alerts,  would 
affect  performance. 
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Appendix  A:  Audio  Alert  Guidelines 

According  to  Deatherage  (1972)  and  Sanders  and  McCormick  (1993),  the 
following  guidelines  should  be  considered  in  designing  to  meet  the  physical  parameters 
of  the  human  ear  and  hearing: 

•  Use  sounds  in  the  200  to  5000Hz  range,  in  particular  the  500  to  3000Hz  because 
this  middle  range  is  the  most  sensitive  region  for  human  hearing. 

•  To  avoid  masking  in  noise,  use  signal  frequencies  different  from  the  noises'  most 
intense  frequencies. 

•  To  capture  attention,  use  modulated  sounds  of  intermittent  beeps  repeating  one 
to  eight  beeps  per  second  or  warbling  sounds  that  vary  between  1  to  3  times  per 
second,  because  these  sounds  rarely  occur  naturally  and  will  capture  operator 
attention. 

•  If  representing  different  conditions,  different  warning  signals  should  be 
discriminable  from  each  other,  and  moderate-intensity  signals  should  be  used. 
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Appendix  B:  Scenario  Events 


Appendix  B  shows  that  the  single-UAV  and  multiple-UAV  scenarios  were 
designed  with  the  same  number  of  events.  The  only  difference  was  that  the  multiple- 
UAV  scenario  has  the  events  divided  over  four  UAVs,  while  the  single-UAV  scenario 
has  them  all  occurring  with  one  UAV.  The  timeline  of  events  for  the  two  scenarios 


(Figure  B-1)  illustrates  this. 
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Figure  B-1:  Major  Events  of  Single-UAV  and  Multiple-UAV  Test  Scenarios. 
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Table  B-1  shows  side-by-side  images  of  the  timeline  display  and  map  display  for 
the  single-UAV  and  multiple-UAV  scenarios.  Again,  as  with  the  timeline  in  Figure  B-1, 
these  displays  show  that  in  both  scenarios  the  operator  has  the  same  number  of  tasks  to 
complete,  and  the  only  difference  between  the  two  scenarios  is  that  for  the  multiple 
scenario,  the  tasks  are  divided  over  four  UAVs  instead  of  just  one  UAV. 


Table  B-1:  Comparison  of  Single-UAV  and  Multiple-UAV  Scenarios. 
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Appendix  C:  Participant  Consent  Form 

CONSENT  TO  PARTICIPATE  IN 
NON-BIOMEDICAL  RESEARCH 

Developing  Deeision  Support  for  Supervisory  Control  of  Multiple  Unmanned  Vehieles 

You  are  asked  to  partieipate  in  a  researeh  study  eonducted  by  Professor  Mary  Cummings  Ph.D, 
from  the  Aeronauties  and  Astronautics  Department  at  the  Massachusetts  Institute  of  Technology 
(M.I.T.).  You  were  selected  as  a  possible  participant  in  this  study  because  the  expected 
population  this  research  will  influence  is  expected  to  contain  men  and  women  between  the  ages 
of  18  and  50  with  an  interest  in  using  computers.  You  should  read  the  information  below,  and 
ask  questions  about  anything  you  do  not  understand,  before  deciding  whether  or  not  to 
participate. 

.  PARTICIPATION  AND  WITHDRAWAL 

Your  participation  in  this  study  is  completely  voluntary  and  you  are  free  to  choose  whether  to  be 
in  it  or  not.  If  you  choose  to  be  in  this  study,  you  may  subsequently  withdraw  from  it  at  any  time 
without  penalty  or  consequences  of  any  kind.  The  investigator  may  withdraw  you  from  this 
research  if  circumstances  arise  which  warrant  doing  so. 

.  PURPOSE  OF  THE  STUDY 

The  study  is  designed  to  evaluate  how  decision  support  tools  or  recommendations,  both  audio 
and  visual,  assist  an  operator  supervising  multiple  simultaneous  dynamic  tasks,  and  how  decision 
support  assistance  and  effectiveness  changes  as  workload  increases.  In  measuring  the 
effectiveness  of  decision  support,  an  operator’s  performance  and  situation  awareness  are  used  as 
metrics.  Situation  awareness  is  generally  defined  as  the  perception  of  the  elements  in  the 
environment,  the  comprehension  of  the  current  situation,  and  the  projection  of  future  status  of  the 
related  system. 

.  PROCEDURES 

If  you  volunteer  to  participate  in  this  study,  we  would  ask  you  to  do  the  following  things: 

•  Attend  a  training  and  practice  session  to  learn  a  video  game-like  software  program  that 
will  have  you  supervising  and  interacting  with  multiple  unmanned  aerial  vehicles 
(estimated  time  0.75  hours). 

•  Practice  on  the  program  will  be  performed  until  an  adequate  level  of  performance  is 
achieved,  which  will  be  determined  by  your  demonstrating  basic  proficiency  in 
monitoring  the  vehicles,  redirecting  them  as  necessary,  executing  commands  such  as 
firing  and  arming  of  payload  at  appropriate  times,  using  decision  support  visualizations 
and/or  recommendations  to  mitigate  timeline  problems,  and  responding  to  radio  calls  by 
clicking  an  acknowledge  button  on  the  software  interface  (estimated  time  0.75  hours). 
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•  Execute  two  thirty  minute  trials  consisting  of  the  same  tasks  as  above  (1  hour) 

•  Attend  a  debriefing  to  determine  your  subjective  responses  and  opinion  of  the  software 
(10  minutes). 

•  Testing  will  take  place  in  MIT  building  37,  room  301 . 

•  Total  time:  2-3  hours,  depending  on  skill  level. 

.  POTENTIAL  RISKS  AND  DISCOMFORTS 

There  are  no  anticipated  physical  or  psychological  risks  in  this  study. 

.  POTENTIAL  BENEFITS 

While  there  is  no  immediate  foreseeable  benefit  to  you  as  a  participant  in  this  study,  your  efforts 
will  provide  critical  insight  into  the  human  cognitive  capabilities  and  limitations  for  people  who 
are  expected  to  supervise  multiple  complex  tasks  at  once,  and  how  decision  support  tools  can 
support  their  task  management. 

.  PAYMENT  FOR  PARTICIPATION 

You  will  be  paid  $  10/hr  to  participate  in  this  study  which  will  be  paid  upon  completion  of  your 
debrief  Should  you  elect  to  withdraw  in  the  middle  of  the  study,  you  will  be  compensated  for  the 
hours  you  spent  in  the  study. 

.  CONFIDENTIALITY 

Any  information  that  is  obtained  in  connection  with  this  study  and  that  can  be  identified  with  you 
will  remain  confidential  and  will  be  disclosed  only  with  your  permission  or  as  required  by  law. 
You  will  be  assigned  a  subject  number  which  will  be  used  on  all  related  documents  to  include 
databases,  summaries  of  results,  etc.  Only  one  master  list  of  subject  names  and  numbers  will 
exist  that  will  remain  only  in  the  custody  of  Professor  Cummings. 

.  IDENTIFICATION  OF  INVESTIGATORS 

If  you  have  any  questions  or  concerns  about  the  research,  please  feel  free  to  contact  the  Principal 
Investigator,  Mary  L.  Cummings,  at  (617)  252-15 12,  e-mail,  missvc@mit.edu,  and  her  address  is 
77  Massachusetts  Avenue,  Room  33-305,  Cambridge,  MA  02139.  The  student  investigators  are 
Hudson  D.  Graham  (719-238-1713,  email:  hgraham@mit.edu),  and  Amy  Brzezinski  (617-276- 
6708,  amybrz@MIT.EDU). 

.  EMERGENCY  CARE  AND  COMPENSATION  FOR  INJURY 

“In  the  unlikely  event  of  physical  injury  resulting  from  participation  in  this  research  you  may 
receive  medical  treatment  from  the  M.I.T.  Medical  Department,  including  emergency  treatment 
and  follow-up  care  as  needed.  Your  insurance  carrier  may  be  billed  for  the  cost  of  such 
treatment.  M.I.T.  does  not  provide  any  other  form  of  compensation  for  injury.  Moreover,  in 
either  providing  or  making  such  medical  care  available  it  does  not  imply  the  injury  is  the  fault  of 
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the  investigator.  Further  information  may  be  obtained  by  calling  the  MIT  Insurance  and  Legal 
Affairs  Office  at  1-617-253-2822.” 


.  RIGHTS  OF  RESEARCH  SUBJECTS 

Y ou  are  not  waiving  any  legal  claims,  rights  or  remedies  because  of  your  participation  in  this 
research  study.  If  you  feel  you  have  been  treated  unfairly,  or  you  have  questions  regarding  your 
rights  as  a  research  subject,  you  may  contact  the  Chairman  of  the  Committee  on  the  Use  of 
Humans  as  Experimental  Subjects,  M.I.T.,  Room  E25-143b,  77  Massachusetts  Ave,  Cambridge, 
MA  02139,  phone  1-617-253-6787. 


SIGNATURE  OF  RESEARCH  SUBJECT  OR  LEGAL  REPRESENTATIVE 


I  understand  the  procedures  described  above.  My  questions  have  been  answered  to  my 
satisfaction,  and  I  agree  to  participate  in  this  study.  I  have  been  given  a  copy  of  this  form. 


Name  of  Subject 


Signature  of  Subject 


Date 


SIGNATURE  OF  INVESTIGATOR 


In  my  judgment  the  subject  is  voluntarily  and  knowingly  giving  informed  consent  and  possesses 
the  legal  capacity  to  give  informed  consent  to  participate  in  this  research  study. 


Signature  of  Investigator 


Date 
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Appendix  D:  Demographics  Survey 


MAUVE-MITUS  Demographic  Survey 


1-  Age: _ 

2.  Gender:  □  Male  □  Female 

3.  Occupation: _ 


If  student: 

a.  Class  Standing:  □  Undergraduate  □  Graduate 

b.  Major: _ 

If  currently  or  formerly  part  of  any  country’s  armed  forces: 

a.  Country /State: _ 

b.  Status:  □  Active  Duty  □  Reserve  □  Retired 

c.  Service:  □  Army  □  Navy  □  Air  Force  □  Other _ 

d.  Rank: _ 

e.  Years  of  Service: _ 

f.  Did  you  ever  serve  in  high  noise  environments?  □  Yes  □  No 

If  yes,  please  explain  what  the  duties  were,  how  long  the  shifts  were,  and  how  many  times  you 

served  these  shifts? 


4.  Do  you  bave  experience  with  remotely  piloted  vebicles  (land,  sea,  air)? 

□  Yes 

□  No 

If  yes: 

a.  Vehicle  type(s)/class(es): 

b.  Number  of  hours: _ 

5.  Do  you  bave  experience  with  radios  sucb  as  those  used  for  communication  in  flying? 

□  Yes 

□  No 


If  yes: 
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Please  explain: 


6.  Have  you  been  to  a  music  concert  in  the  last  Month? 

□  Yes 

□  No 

If  yes: 

a.  Concert  type: _ 

b.  When: _ 

7.  Do  you  have  any  hearing  loss? 

□  Yes 

□  No 

If  yes: 

Please  explain: 


8.  How  often  do  you  play  video  games? 

□  Never 

□  Less  than  1  hour  per  week 

□  Between  1  and  4  hours  per  week 

□  Between  1  and  2  hours  per  day 

□  More  than  2  hours  per  day 

9.  Are  you  color  blind? 

□  Yes 

□  No 

If  yes: 

Whieh  type  of  eolor  blindness  (if  known): 
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Appendix  E:  MAUVE-MITUS  Tutorial 


Spring  2007 


Multi- Aerial  Unmanned  Vehicle 
Experiment  (MAUVE) 
TUTORIAL 


.3 

Hudson  Graham  -  MIT  Humans  and  Automation  Lab  Ulif 

Introduction 


Welcome! 

This  tutorial  is  designed  to  give  you  some  background  on  the  Multi- 
Aerial  Unmanned  Vehicle  Experiment  (MAUVE)  interface  before  you 
arrive  on  testing  day.  Please  take  the  time  to  look  over  the  following 
slides  and  come  prepared  with  questions.  Before  testing  you  will  be 
thoroughly  trained  on  the  actual  interface,  but  being  exposed  to  it 
beforehand  will  be  invaluable  in  speeding  up  this  process. 

Thank  you  in  advance  for  your  participation! 


Experiment  Overview 


I  n  this  experiment,  you  are  an  unmanned  aerial  vehicle  (UAV) 
operator  that  is  responsible  for  supervising  1  to  4  UAVs  collectively 
tasked  with  destroying  a  set  of  time-sensitive  targets  in  a 
suppression  of  enemy  air  defenses  mission.  The  area  contains  enemy 
threats  capable  of  firing  on  your  UAVs. 

The  UAVs  are  highly  autonomous,  and  therefore  only  require  high 
level  mission  execution  from  you.  The  UAVs  launch  with  a  pre¬ 
determined  mission  plan,  so  initial  target  assignments  and  routes 
have  already  been  completed  for  you.  Your  job  will  be  to  monitor 
their  progress,  re-plan  aspects  of  the  mission  in  reaction  to 
unexpected  events,  and  in  some  cases  manually  execute  mission 
critical  actions  such  as  arming  and  firing  of  payloads. 

The  interface  we  have  developed  for  this  experiment  is  called  the 
Multi-Aerial  Unmanned  Vehicle  Experiment  (MAUVE)  and  will  be 
referred  to  by  this  name  from  here  out. 


Objectives 

Your  primary  Directive  in  this  mission  is: 

To  make  sure  the  UAV(s)  maximize  the  number  of  targets  engaged  as  well  as 
arriving  back  to  the  base  safely. 

Supervision  of  the  UAVs  can  be  broken  down  into  the  following 
prioritized  sub- tasks,  from  highest  priority  to  lowest: 

1.  Return  to  base  (RTB)  within  the  time  limit  for  the  mission  (this  limit  will  be 
clearly  marked). 

2.  Comply  with  recovery  rules  for  course  deviations. 

3.  Comply  with  recovery  rules  for  target  late  arrivals. 

4.  Destroy  all  targets  before  their  time  on  target  (TOT)  window  ends. 

5.  Avoid  taking  damage  from  enemies  by  avoiding  all  threat  areas. 

6.  Acknowledge  all  "Push"  radio  calls. 

These  sets  of  objectives  will  often  conflict  with  one  another.  In  these 
cases,  you  must  perform  the  actions  that  have  the  highest  priority 
first. 

Your  performance  will  be  judged  based  on  how  well  you  follow  the  above 
priorities. 


Audio  Alerts 


To  help  you  meet  your  objectives  you  will  receive  auditory  signals  for  both 
course  deviations  and  late  arrivals.  Both  are  induced  by  unanticipated  high 
winds  along  the  planned  flight  path. 

Course  deviations  are  when  a  UAV  is  blown  off  of  the  planned  path.  It  is 

significantly  deviated  when  you  visually  see  that  the  UAV  has  left  the  course  line. 
Deviations  may  occur  over  targets  as  well. 


Late  arrivals  are  when  the  UAV  has  hit  stronger  than  anticipated  head  winds  and 
slows  down.  As  a  result  it  will  now  be  late  to  the  next  target. 

Your  test  proctor  will  provide  further  training  as  to  what  these  auditory 
signals  sound  like  during  the  test  day  training. 

Other  auditory  sounds  to  be  familiar  with  are:  (Note  all  three  are  the  same 
because  all  are  related  to  a  new  message  in  your  message  box.) 

■*  -4  \ 

For  new  messages  in  For  pop-up  threats  For  when  your  UAV  is 

your  message  l^xes  being  fired  upon 

while  flying  through 
a  threat  area. 


Color  Coding 


Throughout  the  displays  you're  about  to  see,  the  following  color  coding  is  used 
to  indicate  each  of  the  5  possible  actions  a  UAV  can  perform  in  MAUVE: 


UAV  Action 

Color 

Enroute 

Gray 

Loitering 

Blue 

Arming  Payload 

Yellow 

Firing  Payload 

Orange 

Return  to  Base 

Green 
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Displays  -  Overview 


During  the  experiment,  you  will  see  two  side-by-side  displays  that  contain 
the  following  major  elements: 

•  Left  Display 

-  Mission  Time 

-  Map  Display 

-  Mission  Execution 

•  Right  Display 

-  Unmanned  Aerial  Vehicle  (UAV)  Status 

-  Decision  Support 

-  Chat  Box 

-  UAV  Health  &  Status  Updates 

The  following  slides  will  ^ow  these  displays  in  detail  and  explain  how  to  use 
them  properly. 


The  three 
major  screen 
elements  on 
the  left  display 
are: 


1 

Mission 

Execution 

2 

Mission 

Time 


3 


Left  Display  -  Overview 


Left  Display  -  Detail 


The  following  slides  detail  all  of  the  elements  contained  on  the  left 
display,  in  this  order: 

*  Map  Display 

*  Mission  Execution 

*  Mission  Time 


Map  Display  -  Detail  -  2 


Naming  Conventions 

•  UAVs 

-  Numbered  1-4 

*  Targets 

-  T-XXP  where  XX  =  target  number  and  P  =  priority 

-  Priority  may  be  High  (H),  Medium  (M),  or  Low  (L) 

-  Exampies: 

•  T-IH  -  Target  1  a  high  priority  target 

•  T-12M  -  Target  12  a  medium  priority  target 

•  T-23L  -  Target  23  a  iow  priority  target 

•  Waypoints  (WP) 

-  WP-XY  where  X  =  UAV#  the  waypoint  is  associated  with  and  Y  =  waypoint 
letter 

-  Examples:  WP-IA,  WP-2C 

*  Threats/ Hazards 

-  H-XXX  where  XXX  =  threat  number 

-  Example:  H-OOl,  H-012 
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Mission  Time  -  Detail 


The  mission  time  display  element  ^ows  the  following: 

1.  Absolute  Time 

-  These  clocks  are  on  the  left  side  of  the  "Mission  Time"  title 

-  Two  formats 

•  Current  Time:  Current  mission  time 

•  EndTime:  End  timeof  the  current  mission 

2.  Relative  Time 

-  These  clocks  are  on  the  right  «de  of  the  "Mission  Hme"  title 

-  Two  formats 

•  Elapsed  Time:  How  long  has  elapsed  since  the  start  of  the  scenario 

•  Remaining  time:  How  long  remains  until  the  mission  is  over 

Example  Mission  Time  Display  Element 

Current:  End:  MISSION  EtBps* *d  R*malnlns: 

I  '  I  I  -  ■  I  ™e  I  .  I  I  I 


Mission  Execution  -  Detail  -  1 


Each  UAV  has  its  own  mission  execution  bar. 

To  bring  it  up  on  the  left  display,  click  anywhere  on  the  desired  UAV's  status 
window  on  the  right  display  OR  on  the  UAV  icon  itself  on  the  left  display. 

Light  green  highlighting  around  the  UAV's  status  bar  and  its  current  mission 
plan  on  the  map  display  tell  you  which  UAV/  route  is  currently  selected 


In  the  display  below  UAV  4  is  highlighted  so  this  is  Ute  mission  execution 
bar  on  the  left  side  of  the  left  dis/^ay. 


Mission  Execution  -  Detail  -  2 


Mission  Execution  Functions 


UAV  4 


Mission  Execution  -  Detaii  -  3 

Mission  Execution  Functions 

4.  Reset  Navigation 


UAV  4 


1.  Arm  Payload 

-  This  button  is  only  enabled  if  the  UAV 
is  selected  while  directly  on  top  of  a 
target,  and  within  the  arming  or 
firing  windows 


2.  Fire  Payload 

-  This  button  is  only  enabled  if  the  UAV 
is  selected  while  directly  on  top  of  a 
target,  armed,  and  within  the  firing 
window  for  that  particular  target 

3.  Skip  Target 

-  This  button  is  used  if  you  decide  to 
skip  a  target  because  you  are  going 
to  be  late  to  it.  It  causes  the  UAV  to 
skip  the  next  target/  waypoint  and 
move  to  the  next  waypoint/  target 


a 


-  Click  anytime  you  have  a  significant 
course  deviation  and  want  to  return  to 
the  planned  course;  it  will  return  the 
UAV  to  the  next  plotted 

waypoint/  target. 

-  Causes  the  UAVto  be  inactive  while 
resetting;  you  may  not  be  able  to  arm 
or  fire  with  that  UAV  when  its 
navigation  system  is  resetting. 

5.  Radio  Monitoring 

-  Radio  chat  from  the  Boston  ground 
control  will  be  playing  and  you  must 
click  the  "Acknowledge  PUSH"  each 
time  "push"  is  called  on  the  radio  chat. 
Hint:  Push  calls  usually  come  in  pairs 
(but  not  always);  the  tower  and an 
aircrew  or  vice  v&sa. 

6.  Target  Assignment  Queue 

-  You  will  not  be  using  this  portion  of  the 
display. 


Right  Display  -  Detail 


The  following  slides  detail  all  of  the  elements  contained  on  the  right 
display,  in  this  order: 

•  UAV  Status 

*  Health  &  Status  Updates 

•  Decision  Support 

*  Chat  Box 
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UAV  Status  -  Detail  -  1 


The  UAV  status  display  shows  the  following 
real-time  information  for  each  UAV: 

•  Status  /  Current  Action 

-This  is  written  out  as  well  as 
represented  by  the  color  of  the  UAV 
icon  to  the  right 

-  For  example:  a  blue  UAV  would  be 
loitering.  This  means  it  has  arrived 
over  the  target  but  is  there  before  the 
arming  and  firing  window  so  is  in  a 
holding  pattern  waiting  for  these 
windows. 

•  Current  Target  Name 

•  Position  in  Latitude  &  Longitude 

-You  will  not  need  this  in  the  scenario. 

-Given  in  degrees,  minutes,  and  seconds 

•  Altitude 

-  This  is  a  static  number  that  is  not  used 
in  the  scenario  in  any  way 


UAV  Status  -  Detail  -  2 


The  UAV  status  display  ^ows  the  following  real¬ 
time  information  for  each  UAV: 


•  Course 


Example  UAV  Status  Display  Element 


-You  will  not  need  this  during  the  scenario. 

-0°  indicates  due  north;  increases  in  a 
clockwise  manner 

•  Speed 

-  The  UAVs  are  set  to  travel  at  a  constant 
speed  of  200  kts. 

- 1  f  there  is  significant  head  wind  the  UAV 
may  slow  down.  This  can  precipitate  late 
target  arrivals 

•  Payload  Ready 

-  This  reflects  whether  the  UAV  has  a 
payload  ready  for  the  current  target 

-  Will  say  'Yes"  if  the  UAV  is  armed,  “No”  if 
not 


(continued  on  next  ^ide) 


UAV  Health  &  Status  Updates  -  Detail 


The  Health  &  Status  Updates  box  contains  messages  horn  ^lecific  UAVs  intoided  to 
inform  the  operator.  Messages  are  color  coded  as  follows: 

*  Red  ss  UAV  Health  messages 

-  UAV  is  under  fire  from  a  threat 

-  Again,  a  standard  audio  alert  will  play  when  you  recave  red  messages. 

*  Bold  Black  s  UAV  Status  messages,  action  required 

-  UAV  is  available  to  arm  or  fire 

*  Black  s  UAV  Status  messages,  no  action  required 

-  UAV  has  completed  arming  or  firing 

Example  Health  &  Status  Updates  Window 

t  StMut  UpdtM* 

LAVI  IMsrvrtaerwH'HI 

I'M’  I  a:*  at  UMa  rm  H-MI 

t'M'  I  ar«*  «n  -•  'Mv  r**  ha  n  ct; 

1;AV  «  02  U  J»t  •  '  AnMNt  WWB  t«  iwiM  T  4M 
UkV  i  02.1). lit  to  T  4M 

CAV  4  02.U  45t  ht  T  4M 

UAV4  02:U:«*t  •  .li'JOfc  » IW*  hr  T-4M 


Decision  Support  -  Remember  the  Color  Coding 


Color  coding  is  an  important  element  of  the  deci^on  support,  so  take  a  look  at 
it  again! 


UAV  Action 

Color 

Enroute 

Gray 

Loitering 

Blue 

Arming  Payload 

Yellow 

Firing  Payload 

Orange 

Return  to  Base 

Green 

Decision  Support  -  Detail  -  1 


The  active  level  of  decision  support  contains  a  visual  representation  of  what  and 
whm  targets  are  approaching  through  a  relative  timeline  and  projective 
decision  support  display  for  each  UAV. 

•  The  arming  and  firing  windows  cannot  by  changed  solely  at  the  will  of  the 
the  operator,  i.e.  Operators  may  request  time  on  target  (TOT)  delay 
requests,  but  must  get  approval  before  the  arming  and  firing  window  will 
be  moved  back  (if  approved). 


Example  Active  Decision  Support  Window 


^  ■  - 

t  r  r  r  :  — 

'“'Tr  ■  - 

'7^'  '  - 

Decision  Support  -  Detail  -  2 


Arming  and  firing  el^nents  are  color  coded  in  the  same  way  as  corresponding 
UAV  actions.  For  each  target  the  following  information  is  represented: 

1.  Arming  Window  =  Yellow 

-  10  seconds  long  and  takes  approximately  3-7  seconds  to  arm. 

-  Payload  for  the  relevant  target  may  be  armed,  but  not  fired  during 
this  time 

-  Always  occurs  immediately  before  the  firing  window 

2.  TOT/  Firing  Window  =  Orange 

-  20  seconds  long  and  takes  approximately  3-7  seconds  to  fire. 

-  Payload  must  be  fired  at  file  relevant  target  during  fills  time 

-  I  n  addition  to  the  arming  window,  a  payload  may  also  be  armed 
during  this  time 

-  Target  name  is  printed  vertically  in  the  center  of  the  window, 
(M'iority  is  print^  above  the  window. 
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Decision  Support  -  Detail  -  3 


Mission  planning  information  reflects  when  UAVs  will  reach  important  points 
in  the  scenario,  such  as: 

1.  Waypoints/  Loiterpoints/  Base  =  Black  Triangles 

-  Names  are  printed  above  the  relevant  UAV's  timeline 

2.  UAV  Arrival  at  Targets  =  Black  Rectangles 

-  Names  are  printed  below  the  relevant  UAV’s  timeline 

-  Note  that  each  target  name  will  appear  twice  on  the  timeline, 
once  for  when  the  UAV  will  arrive  at  that  target  and  once  at  the 
center  of  that  target’s  firing  window 

3.  Late  UAV  Arrival  =  Red  Rectangles 

-  Black  Rectangle  turns  red  and  moves  past  the  target  to  when  the 
UAV  will  arrive. 


Decision  Support  -  Detail  -  4 


The  active  level  of  decision  support  aids  the  user  by  identifying  possible  late 
target  arrivals  and  areas  of  potential  high  workload  on  the  timeline. 

•  A  late  target  arrival  is  defined  as  when  a  UAV  will  arrive  to  a  target  after 
its  scheduled  time  on  target 

•  Corrective  actions  for  late  arrivals  are  based  on  the  priority  of  the  target 
you  are  projected  to  be  late  to.  For: 

-  Low  priority  targets,  skip  them  by  clicking  “Skip  Target"  on  the 
navigation  screen. 

-  Medium  priority  targets,  either  skip  them  or  use  the  decision  support 
visualization  (DSV)  to  possibly  request  a  delay.  Remember  your 
priorities  of  wanting  to  hit  all  the  targets.  See  the  nexthyursHe/es  for 
an  explanation  of  how  to  use  the  DSV  (shown  below). 

-  High  priority  targets,  use  the  DSV  and  then  decide  whether  to  request 
a  TOT  delay  or  to  skip  the  target  by  clicking  "Skip  Target" 

Note:  The  corrective  actions  above  should  only  be  taken  when  a  iate  arrival  is 
projected  by  the  red  rectangle  on  the  screen  and  your  audio. 
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Decision  Support  -  Detail  -  5 


The  decision  support  visuaiization  (DSV)  helps  the  user  manage  the 
scheduie  by  showing  timeiine  issues  and  projecting  “what  if" 
conditions  of  the  effects  on  the  timeiine  tosed  upon  user 
decisions. 

•  Each  UAV's  DSV  is  uses  emergent  features  to  show  problems 
that  currently  exist  or  that  may  exist  if  a  TOT  delay  is  given. 

•  No  issues  (late  target  arrivals)  are  indicated  by  no  rectangles 
being  displayed. 

-  The  picture  below  shows  the  DSV  for  when  there  is  no  late 
arrivals.  It  depicts  this  by  having  no  rectangles  above  the 
line  on  the  left  side  of  the  display  in  the  “Late  Arrival” 
section. 

Note:  The  DSV  will  be  inactive  except  for  when  you  are  going  to  be 
iate  a  medium  or  high  priority  target. 


Decision  Support  -  Detail  -  6 


Late  arrivals  are  represented  by  a  rectangle  occurring  on  the 
DSV  above  ti>e  line  on  the  left  side  of  the  display  in  the  "Late 
Arrivals”  section.  It  will  also  be  highlighted  yellow  as  it  is 
below. 

-  The  target's  priority  is  indicated  within  the  rectangle  and 
by  the  rectangle’s  size.  The  higher  the  priority  of  the 
target  that  the  UAV  will  be  late  for  the  taller  the  rectangle 
will  be. 


Late  Arrivals 

n  .  .  .  . 

ML 

T-2M 

- 

1  Req.  Delay 

Probability; 

1  NONE  1 

Projection 

Decision  Support  -  Detail  -  7 


•  Below  the  center  line  is  for  the  “what  if”  condition;  after  the 
user  selects  a  target  that  they  might  request  a  TOT  delay  for  it 
will  show  the  projected  future  late  arrivals  for  that  UAV  below 
the  centerline  on  the  left  side  of  the  display  in  the  “Projection” 
section. 


-  The  example  below  shows  that  if  a  TOT  delay  request  is 
granted  for  target  T-16H,  the  UAV  will  then  be  late  to  a 
Low  Priority  target. 

-  The  Probability  in  the  bottom  right  of  this  display  shows 
the  likelihood  of  a  TOT  delay  being  granted.  The  further  in 
advance  a  delay  is  requested  the  higher  the  likelihood  of  it 
being  granted.  Do  not  request  a  delay  again  if  your  first 
request  for  a  delay  on  that  target  is  denied. 


Late  Arrivals 


Projection 


T  16H 

T-14L 

M1M 

Probability; 

T-8L 

FmedI  I 

Decision  Support  -  Detail  -  8 


Each  UAV  possesses  a  DSV  display  to  help  the 
user  understand  the  potential  effects  of 
deci»ons 

•  A  list  of  all  the  mission  targets  on  the  timeline 
appears  to  the  right  of  each  UAV's  DSV  display. 

■  I  n  using  the  display  to  the  right  for  the  top  UAV: 

-  The  user  is  considering  requesting  a  delay 
forT-7H,  a  high  priority  target-  However, 
it  shows  they  will  now  be  late  to  another 
high  priority  target  even  with  this  delay. 
This  is  where  you,  as  the  user,  will  have  to 
make  a  value  call.  Do  you  request  a  delay 
or  skip  the  target  by  clicking  "Skip  Target?" 
Most  of  the  time  you  will  not  want  to 
request  a  delay  if  you  know  it  is  going  to 
create  another  delay. 

■  I  n  using  the  display  to  the  right  for  the  second 
UAV: 

-  The  user  is  considering  requesting  a  delay 
for  T-16H,  a  high  priority  target.  This  will 
result  in  the  UAV  being  late  to  a  low 
priority  target.  So  in  this  case  you,  as  the 
user,  would  want  to  request  this  delay 
because  it  means  you  can  hit  the  high 
priority  target  for  the  trade-off  of  now 
missing  a  low  priority  target. 
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Chat  Box  -  Detail 


The  chat  box  contains  a  time  history  of  all  Au/na/;  interactions.  These  interactions  are 
color  coded  as  follows: 


‘  Red  a  I  nt^ligence  updates 

-  Again,  a  standard  audio  alert  will  play  when  you  receive  red  messages. 

*  Blacks  Message  to/ from  Base,  no  response  required 

-  Messages  that  inform  you,  but  do  not  require  a  specific  response 

The  chat  box  is  purely  informational.  It  will  provide  you  with  updates,  but  you  will  not 
input  anything  in  the  chat  box. 


Example  Message  History  Window 

oj<»ja~wawi.i>iisip« 

or «  it)  •  • 

•w  A1 «  90  —  (UpM  •  MK  M  rOT  *■  t  M  «« 
iiM<i:«i>i>»>evTOTiarmi»uuMC  uoae 


Conclusion 


You  are  now  ready  to  proceed  to  hands-on  training  with  the  MAUVE 
interface.  Remember  to  bring  any  questions  you  have  to  the 
experimenter  on  testing  day! 
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Appendix  F:  Post-Experiment  Survey 


MAUVE  Post-Test  Feedback 

1.  How  did  the  audio  cues  help  or  hinder  you  in  managing  late-arrivalsl 


2.  How  did  the  audio  cues  help  or  hinder  you  in  managing  course-deviations? 


3.  How  would  you  change  the  audio  cues  for  the  iate-arrivais  to  have  them 
better  aid  you  during  your  mission? 
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4.  How  would  you  change  the  audio  cues  for  the  course-deviations  to  have 
them  better  aid  you  during  your  mission? 


5.  In  general  how  would  you  change  all  the  audio  to  help  you  with  the  control 
during  your  mission? 


6.  Please  express  any  other  comments  you  may  like  to  share: 
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Appendix  G:  GLM  Analysis:  SPSS  OUTPUT 


Included  in  this  appendix  are  the  key  SPSS  outputs  for  the  data  analysis. 


G.l.  Course-deviation  Reaction  Times  (for  4  audio  conditions) 


Met  normality  and  homogeneity  assumptions. 


Table  G-1:  Course-deviation  Reaction  Times  (4  audio  conditions)  Within-Subjects  Contrasts. 


Type  III  Sum 

Source 

Scenario 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Scenario 

Linear 

14.029 

1 

14.029 

3.215 

.082 

scenario  * 

AudioScheme 

Linear 

10.780 

3 

3.593 

.824 

.490 

Error(scenario) 

Linear 

152.711 

35 

4.363 

Table  G-2:  Course-deviation  Reaction  Times  (4  audio  conditions)  Between-Subjects  Effects. 


Source 

Type  III  Sum 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Intercept 

8998.235 

1 

8998.235 

1089.907 

.000 

Audio_Scheme 

71.289 

3 

23.763 

2.878 

.050 

Error 

288.959 

35 

8.256 
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Table  G-3:  Course-deviation  Reaction  Times  (4  audio  conditions)  Tukey  Test  Comparisons. 


(1) 

Audio_Scheme 

(J)  AudioScheme 

Mean 

Differen 

ce  (l-J) 

Std. 

Error 

Sig. 

95%  Confidence  Interval 

Lower  Bound 

Upper  Bound 

BothCont 

LateCont 

-1.6292 

.90862 

.294 

-4.0796 

.8213 

DevCont 

.3667 

.90862 

.977 

-2.0838 

2.8171 

BothThresh 

-1.7949 

.93352 

.237 

-4.3125 

.7227 

LateCont 

BothCont 

1.6292 

.90862 

.294 

-.8213 

4.0796 

DevCont 

1.9958 

.90862 

.144 

-.4546 

4.4463 

BothThresh 

-.1657 

.93352 

.998 

-2.6834 

2.3519 

DevCont 

BothCont 

-.3667 

.90862 

.977 

-2.8171 

2.0838 

LateCont 

-1.9958 

.90862 

.144 

-4.4463 

.4546 

BothThresh 

-2.1616 

.93352 

.114 

-4.6792 

.3560 

BothThresh 

BothCont 

1.7949 

.93352 

.237 

-.7227 

4.3125 

LateCont 

.1657 

.93352 

.998 

-2.3519 

2.6834 

DevCont 

2.1616 

.93352 

.114 

-.3560 

4.6792 

G.2.  Course-deviation  Reaction  Times  (for  2  audio  alerts) 


Met  normality  and  homogeneity  assumptions. 


Table  G-4:  Course-deviation  Reaction  Times  (2  audio  alerts)  Within-Subjects  Contrast. 


Type  III  Sum 

Source 

Scenario 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Scenario 

Linear 

14.534 

1 

14.534 

3.311 

.077 

Scenario  * 

Audio_Scheme 

Linear 

1.089 

1 

1.089 

.248 

.621 

Error(scenario) 

Linear 

162.403 

37 

4.389 
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Table  G-5:  Course-deviation  Reaction  Times  (2  audio  alerts)  Between-Subjects  Effects. 


Source 

Type  III  Sum 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Intercept 

9007.397 

1 

9007.397 

1146.991 

.000 

Audio_Scheme 

69.684 

1 

69.684 

8.874 

.005 

Error 

290.564 

37 

7.853 

G.3.  Transformed  (natural  log)  Late-arrival  Reaction  Times 


Met  normality  and  homogeneity  assumptions  after  original  data  was  transformed  with 
a  natural  log  transformation. 


Table  G-6:  Transformed  (natural  log)  Late-arrival  Reaction  Times  Within-Subjects  Contrasts. 


Type  III  Sum 

Source 

Scenario 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Scenario 

Linear 

6.513 

1 

6.513 

20.737 

.000 

scenario  * 

AudioScheme 

Linear 

1.619 

3 

.540 

1.719 

.181 

Error(scenario) 

Linear 

10.992 

35 

.314 

Table  G-7:  Transformed  (natural  log)  Late-arrival  Reaction  Times  Between-Subjects  Effects. 


Source 

Type  III  Sum 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Intercept 

244.837 

1 

244.837 

305.851 

.000 

Audio_Scheme 

8.028 

3 

2.676 

3.343 

.030 

Error 

28.018 

35 

.801 
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Table  G-8:  Transformed  (natural  log)  Late-arrival  Reaction  Times  Tukey  Test  Comparisons. 


(1)  Audio_Scheme 

(J)  AudioScheme 

Mean 

Difference 

(l-J) 

Std. 

Error 

Sig. 

95%  Confidence  Interval 

Lower  Bound 

Upper  Bound 

BothCont 

LateCont 

.0092 

.28293 

1.000 

-.7539 

.7722 

DevCont 

-.7099 

.28293 

.076 

-1.4729 

.0532 

BothThresh 

.0633 

.29069 

.996 

-.7206 

.8473 

LateCont 

BothCont 

-.0092 

.28293 

1.000 

-.7722 

.7539 

DevCont 

-.7191 

.28293 

.071 

-1.4821 

.0440 

BothThresh 

.0542 

.29069 

.998 

-.7298 

.8381 

DevCont 

BothCont 

.7099 

.28293 

.076 

-.0532 

1.4729 

LateCont 

.7191 

.28293 

.071 

-.0440 

1.4821 

BothThresh 

.7732 

.29069 

.054 

-.0107 

1.5572 

BothThresh 

BothCont 

-.0633 

.29069 

.996 

-.8473 

.7206 

LateCont 

-.0542 

.29069 

.998 

-.8381 

.7298 

DevCont 

-.7732 

.29069 

.054 

-1.5572 

.0107 

G.4.  Transformed  (natural  log)  Late-arrival  Reaction  Times  (with 
BothCont/LateCont/BothThresh  Combined  against  DevCont) 


Met  normality  and  homogeneity  assumptions  after  original  data  was  transformed  with 
a  natural  log  transformation. 


Table  G-9:  Transformed  (natural  log)  Late-arrival  Reaction  Times  (with 
BothCont/LateCont/BothThresh  Combined  against  DevCont)  Within-Subjects  Contrasts. 


Type  III  Sum 

Source 

Scenario 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Scenario 

Linear 

7.476 

1 

7.476 

23.730 

.000 

scenario  * 

AudioScheme 

Linear 

.954 

1 

.954 

3.029 

.090 

Error(scenario) 

Linear 

11.657 

37 

.315 
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Table  G-10:  Transformed  (natural  log)  Late-arrival  Reaction  Times  (with 
BothCont/LateCont/BothThresh  Combined  against  DevCont)  Between-Subjects  Effects. 


Source 

Type  III  Sum 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Intercept 

228.000 

1 

228.000 

300.624 

.000 

Audio_Scheme 

7.984 

1 

7.984 

10.528 

.002 

Error 

28.062 

37 

.758 

G.5.  NASA  TLX  Scores 


Met  normality  and  homogeneity  assumptions. 


Table  G-11:  NASA  TLX  Scores  Within-Subjects  Contrasts. 


Type  III  Sum 

Source 

Scenario 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Scenario 

Linear 

3.364 

1 

3.364 

.058 

.811 

Scenario  * 

AudioScheme 

Linear 

215.197 

3 

71.732 

1.234 

.312 

Error(scenario) 

Linear 

2035.166 

35 

58.148 

Table  G-12:  NASA  TLX  Between-Subjects  Effects. 


Source 

Type  III  Sum 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Intercept 

96127.322 

1 

96127.322 

188.194 

.000 

Audio_Scheme 

1244.548 

3 

414.849 

.812 

.496 

Error 

17877.551 

35 

510.787 
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G.6.  Missed  Radio  Calls 


Met  normality  and  homogeneity  assumptions. 


Table  G-13:  Missed  Radio  Calls  Within-Subjects  Contrasts. 


Type  III  Sum 

Source 

Scenario 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Scenario 

Linear 

34.307 

1 

34.307 

1.411 

.243 

scenario  * 

AudioScheme 

Linear 

65.774 

3 

21.925 

.902 

.450 

Error(scenario) 

Linear 

850.944 

35 

24.313 

Table  G-14:  Missed  Radio  Calls  Between-Subjects  Effects. 


Source 

Type  III  Sum 

of  Squares 

Df 

Mean  Square 

F 

Sig. 

Intercept 

7966.465 

1 

7966.465 

92.043 

.000 

Audio_Scheme 

210.033 

3 

70.011 

.809 

.498 

Error 

3029.300 

35 

86.551 
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